2021-05-12T08:51:51.934 INFO:root:teuthology version: 1.1.0-19220a3b 2021-05-12T08:51:51.936 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/kchai-2021-05-12_08:37:28-rados-wip-kefu-testing-2021-05-12-1357-distro-basic-smithi/6111308 --name kchai-2021-05-12_08:37:28-rados-wip-kefu-testing-2021-05-12-1357-distro-basic-smithi --owner scheduled_kchai@teuthology --description rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/osd} -- /home/teuthworker/archive/kchai-2021-05-12_08:37:28-rados-wip-kefu-testing-2021-05-12-1357-distro-basic-smithi/6111308/orig.config.yaml 2021-05-12T08:51:51.970 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-05-12T08:51:52.030 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2021-05-12_08:37:28-rados-wip-kefu-testing-2021-05-12-1357-distro-basic-smithi/6111308 branch: wip-kefu-testing-2021-05-12-1357 description: rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/osd} email: null first_in_suite: false job_id: '6111308' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2021-05-12_08:37:28-rados-wip-kefu-testing-2021-05-12-1357-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.3' overrides: admin_socket: branch: wip-kefu-testing-2021-05-12-1357 ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: a8c673d555488beec9ccda5812ba822e69c12ea9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: a8c673d555488beec9ccda5812ba822e69c12ea9 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-kefu-testing-2021-05-05-1449 sha1: 5cd0ad0001da189274219f5e829752717eb856d2 owner: scheduled_kchai@teuthology priority: 60 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: a8c673d555488beec9ccda5812ba822e69c12ea9 sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2021-05-05-1449 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5cd0ad0001da189274219f5e829752717eb856d2/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5cd0ad0001da189274219f5e829752717eb856d2 targets: smithi028.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCw0QMKTwZQlo8HxIJa2IwRpHp3CrHOKLHMFQrNFhlDMrnohcUuUUfX9sKVhi4AkwdefCoe0psR8CUx6CJruXX+9cndoAcafnU1kTOBWyt7rCaFKfwPsS8VJWImFfWsyIYgJCtJ4/VED08DAzKxFABRo20bCSQ+Iat/AvKzR5+jWaTHyWnz+pbYkJ+gvT/XQUVnNHBlezzoxGlVTxe0r5OOANliesCDww1VCGfpkrCqhzPmQWEuTyg24clMcDIf+SltOyGJRRM2cdo1ktDuaagQGQl6Mfb02IFAkJKsC/U/bxrIgE5EEvaiETTc8em1RsJBWb7uvWzd6pZLFVDf4cPFDTRJn65BBE+peBGlcAX8vULq4NMRqm/axmNcRGBGtUm7KrGk3J9ypLpSbMN1bzrCYABLXVQ9F+dSbfu1m8Ir+jtk6P8wn9LfCGKtyx911uYdue32zNiKqOJ5Bi3yFw1Dx+ACIXLpDrGqJHiPURq1eFQ5gLr5UaMZOAEGvCycjk0= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master teuthology_sha1: 19220a3bd6e252c6e8260827019668a766d85490 timestamp: 2021-05-12_08:37:28 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.8274 2021-05-12T08:51:52.030 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5cd0ad0001da189274219f5e829752717eb856d2/qa; will attempt to use it 2021-05-12T08:51:52.032 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5cd0ad0001da189274219f5e829752717eb856d2/qa/tasks 2021-05-12T08:51:52.032 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-05-12T08:51:52.036 INFO:teuthology.task.internal:Checking packages... 2021-05-12T08:51:52.064 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'a8c673d555488beec9ccda5812ba822e69c12ea9' 2021-05-12T08:51:52.064 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-05-12T08:51:52.065 INFO:teuthology.packaging:ref: None 2021-05-12T08:51:52.065 INFO:teuthology.packaging:tag: None 2021-05-12T08:51:52.065 INFO:teuthology.packaging:branch: wip-kefu-testing-2021-05-12-1357 2021-05-12T08:51:52.065 INFO:teuthology.packaging:sha1: a8c673d555488beec9ccda5812ba822e69c12ea9 2021-05-12T08:51:52.066 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-kefu-testing-2021-05-12-1357 2021-05-12T08:51:52.230 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-4095.ga8c673d5 2021-05-12T08:51:52.230 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-05-12T08:51:52.244 INFO:teuthology.task.internal:no buildpackages task found 2021-05-12T08:51:52.245 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-05-12T08:51:52.305 INFO:teuthology.task.internal:Saving configuration 2021-05-12T08:51:52.315 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-05-12T08:51:52.383 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-12T08:51:52.406 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-05-12 08:47:46.551446', 'description': '/home/teuthworker/archive/kchai-2021-05-12_08:37:28-rados-wip-kefu-testing-2021-05-12-1357-distro-basic-smithi/6111308', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/JLNvkvlGGQZBaQdJtZV7PgslV6RynjMk5ZchlJp/Zl2fsz/JhHlXlhqac7BO9B8kwl2Lb4WTBtio+0dEXokk=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi028.front.sepia.ceph.com'} 2021-05-12T08:51:52.407 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-05-12T08:51:52.469 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2021-05-12T08:51:52.469 INFO:teuthology.run_tasks:Running task console_log... 2021-05-12T08:51:52.578 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe3bfc13ea0>, signals=[15]) 2021-05-12T08:51:52.580 INFO:teuthology.run_tasks:Running task internal.connect... 2021-05-12T08:51:52.626 INFO:teuthology.task.internal:Opening connections... 2021-05-12T08:51:52.627 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2021-05-12T08:51:52.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T08:51:52.704 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-05-12T08:51:52.721 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2021-05-12T08:51:52.881 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2021-05-12T08:51:52.882 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2021-05-12T08:51:52.936 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Linux" 2021-05-12T08:51:52.936 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="8" 2021-05-12T08:51:52.937 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2021-05-12T08:51:52.937 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2021-05-12T08:51:52.937 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="8" 2021-05-12T08:51:52.937 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el8" 2021-05-12T08:51:52.938 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-12T08:51:52.938 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2021-05-12T08:51:52.938 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-12T08:51:52.938 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2021-05-12T08:51:52.938 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-12T08:51:52.939 INFO:teuthology.orchestra.run.smithi028.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-12T08:51:52.939 INFO:teuthology.orchestra.run.smithi028.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-12T08:51:52.940 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2021-05-12T08:51:52.962 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-05-12T08:51:52.974 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-05-12T08:51:53.054 INFO:teuthology.task.internal:Checking for old test directory... 2021-05-12T08:51:53.055 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2021-05-12T08:51:53.071 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-05-12T08:51:53.150 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-05-12T08:51:53.150 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2021-05-12T08:51:53.179 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-05-12T08:51:53.294 INFO:teuthology.run_tasks:Running task kernel... 2021-05-12T08:51:53.366 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}} 2021-05-12T08:51:53.368 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-05-12T08:51:53.368 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2021-05-12T08:51:53.386 INFO:teuthology.orchestra.run.smithi028.stdout:4.18.0-240.10.1.el8_3.x86_64 2021-05-12T08:51:53.391 INFO:teuthology.task.kernel:Running kernel on smithi028: 4.18.0-240.10.1.el8_3.x86_64 2021-05-12T08:51:53.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2021-05-12T08:51:56.997 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-8 - AppStream 7.3 MB/s | 6.3 MB 00:00 2021-05-12T08:51:58.157 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-8 - Base 25 MB/s | 2.3 MB 00:00 2021-05-12T08:51:58.615 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-8 - Extras 144 kB/s | 9.6 kB 00:00 2021-05-12T08:51:59.485 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Linux 8 - AppStream 7.7 MB/s | 6.3 MB 00:00 2021-05-12T08:52:00.938 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Linux 8 - BaseOS 6.8 MB/s | 2.3 MB 00:00 2021-05-12T08:52:01.533 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Linux 8 - Extras 50 kB/s | 9.6 kB 00:00 2021-05-12T08:52:03.075 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 6.3 MB/s | 9.4 MB 00:01 2021-05-12T08:52:05.217 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2021-05-12T08:52:06.817 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-4.18.0-240.10.1.el8_3.x86_64 is already installed. 2021-05-12T08:52:06.817 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-05-12T08:52:07.054 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T08:52:07.063 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T08:52:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2021-05-12T08:52:07.064 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T08:52:07.064 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T08:52:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 4.4 M 2021-05-12T08:52:07.065 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2021-05-12T08:52:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 30 M 2021-05-12T08:52:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 26 M 2021-05-12T08:52:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:52:07.065 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T08:52:07.066 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T08:52:07.066 INFO:teuthology.orchestra.run.smithi028.stdout:Install 3 Packages 2021-05-12T08:52:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:52:07.066 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 60 M 2021-05-12T08:52:07.066 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 83 M 2021-05-12T08:52:07.067 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T08:52:07.239 INFO:teuthology.orchestra.run.smithi028.stdout:(1/3): kernel-4.18.0-240.22.1.el8_3.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2021-05-12T08:52:07.757 INFO:teuthology.orchestra.run.smithi028.stdout:(2/3): kernel-core-4.18.0-240.22.1.el8_3.x86_64 44 MB/s | 30 MB 00:00 2021-05-12T08:52:08.124 INFO:teuthology.orchestra.run.smithi028.stdout:(3/3): kernel-modules-4.18.0-240.22.1.el8_3.x86 24 MB/s | 26 MB 00:01 2021-05-12T08:52:08.127 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T08:52:08.127 INFO:teuthology.orchestra.run.smithi028.stdout:Total 57 MB/s | 60 MB 00:01 2021-05-12T08:52:08.217 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T08:52:08.549 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T08:52:08.549 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T08:52:09.702 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T08:52:09.720 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T08:52:12.129 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T08:52:12.571 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-core-4.18.0-240.22.1.el8_3.x86_64 1/3 2021-05-12T08:52:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-4.18.0-240.22.1.el8_3.x86_64 1/3 2021-05-12T08:52:16.203 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-4.18.0-240.22.1.el8_3.x86_64 2/3 2021-05-12T08:52:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-4.18.0-240.22.1.el8_3.x86_64 2/3 2021-05-12T08:52:19.465 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-12T08:52:41.244 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-12T08:52:41.597 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-12T08:52:41.598 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-4.18.0-240.22.1.el8_3.x86_64 1/3 2021-05-12T08:52:41.598 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-core-4.18.0-240.22.1.el8_3.x86_64 2/3 2021-05-12T08:52:41.903 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-4.18.0-240.22.1.el8_3.x86_64 3/3 2021-05-12T08:52:41.904 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T08:52:42.296 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:52:42.296 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T08:52:42.296 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-4.18.0-240.22.1.el8_3.x86_64 2021-05-12T08:52:42.297 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core-4.18.0-240.22.1.el8_3.x86_64 2021-05-12T08:52:42.297 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-4.18.0-240.22.1.el8_3.x86_64 2021-05-12T08:52:42.297 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:52:42.297 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T08:52:42.385 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel --last | head -n 1 2021-05-12T08:52:42.516 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-4.18.0-240.22.1.el8_3.x86_64 Wed 12 May 2021 08:52:19 AM UTC 2021-05-12T08:52:42.518 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.22.1.el8_3.x86_64 2021-05-12T08:52:42.518 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-240.10.1.el8_3.x86_64 Expected: 4.18.0-240.22.1.el8_3.x86_64 2021-05-12T08:52:42.519 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-05-12T08:52:42.519 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-05-12T08:52:42.519 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2021-05-12T08:52:42.519 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2021-05-12T08:52:42.520 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel --last | head -n 1 2021-05-12T08:52:42.633 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-4.18.0-240.22.1.el8_3.x86_64 Wed 12 May 2021 08:52:19 AM UTC 2021-05-12T08:52:42.634 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.22.1.el8_3.x86_64 2021-05-12T08:52:42.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2021-05-12T08:52:42.666 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2021-05-12T08:52:42.667 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2021-05-12T08:52:42.667 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.02 2021-05-12T08:52:42.667 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 90.el8 2021-05-12T08:52:42.668 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2021-05-12T08:52:42.668 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Mon 01 Mar 2021 06:17:56 PM UTC 2021-05-12T08:52:42.668 INFO:teuthology.orchestra.run.smithi028.stdout:Group : System Environment/Base 2021-05-12T08:52:42.668 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 9197207 2021-05-12T08:52:42.668 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2021-05-12T08:52:42.669 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Tue 08 Sep 2020 07:14:39 PM UTC, Key ID 05b555b38483c65d 2021-05-12T08:52:42.669 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.02-90.el8.src.rpm 2021-05-12T08:52:42.669 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Tue 08 Sep 2020 07:00:29 PM UTC 2021-05-12T08:52:42.669 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : kbuilder.bsys.centos.org 2021-05-12T08:52:42.670 INFO:teuthology.orchestra.run.smithi028.stdout:Relocations : (not relocatable) 2021-05-12T08:52:42.670 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : CentOS BuildSystem 2021-05-12T08:52:42.670 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2021-05-12T08:52:42.670 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2021-05-12T08:52:42.670 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2021-05-12T08:52:42.671 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2021-05-12T08:52:42.671 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:52:42.671 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-05-12T08:52:42.672 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2021-05-12T08:52:42.672 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2021-05-12T08:52:42.672 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2021-05-12T08:52:42.672 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:52:42.672 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2021-05-12T08:52:42.673 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-05-12T08:52:42.674 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 4.18.0-240.22.1.el8_3.x86_64 2021-05-12T08:52:42.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-05-12T08:52:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2021-05-12T08:52:46.255 INFO:teuthology.orchestra.run.smithi028.stderr:done 2021-05-12T08:52:46.257 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2021-05-12T08:52:46.296 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.uNK1yJ6P5q 2021-05-12T08:52:46.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uNK1yJ6P5q 2021-05-12T08:52:46.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.uNK1yJ6P5q 2021-05-12T08:52:46.514 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.uNK1yJ6P5q is 6KB 2021-05-12T08:52:46.568 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.uNK1yJ6P5q 2021-05-12T08:52:46.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2021-05-12T08:52:46.658 INFO:teuthology.orchestra.run.smithi028.stdout:631d84db90f7409fbdd390573ae89718-0-rescue.conf 2021-05-12T08:52:46.658 INFO:teuthology.orchestra.run.smithi028.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.10.1.el8_3.x86_64.conf 2021-05-12T08:52:46.658 INFO:teuthology.orchestra.run.smithi028.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.22.1.el8_3.x86_64.conf 2021-05-12T08:52:46.659 INFO:teuthology.orchestra.run.smithi028.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.el8.x86_64.conf 2021-05-12T08:52:46.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default 631d84db90f7409fbdd390573ae89718-4.18.0-240.22.1.el8_3.x86_64 2021-05-12T08:52:46.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2021-05-12T08:53:16.863 INFO:teuthology.misc:Re-opening connections... 2021-05-12T08:53:16.863 INFO:teuthology.misc:trying to connect to ubuntu@smithi028.front.sepia.ceph.com 2021-05-12T08:53:16.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T08:53:16.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T08:54:16.927 DEBUG:teuthology.orchestra.remote:timed out 2021-05-12T08:54:16.929 DEBUG:teuthology.misc:waited 60.06532669067383 2021-05-12T08:54:17.930 INFO:teuthology.misc:trying to connect to ubuntu@smithi028.front.sepia.ceph.com 2021-05-12T08:54:17.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T08:54:17.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T08:54:18.182 DEBUG:teuthology.orchestra.run.smithi028:> true 2021-05-12T08:54:18.493 DEBUG:teuthology.misc:waited 61.629469871520996 2021-05-12T08:54:19.494 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2021-05-12T08:54:19.495 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-240.22.1.el8_3.x86_64"... 2021-05-12T08:54:19.495 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2021-05-12T08:54:19.511 INFO:teuthology.orchestra.run.smithi028.stdout:4.18.0-240.22.1.el8_3.x86_64 2021-05-12T08:54:19.512 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-240.22.1.el8_3.x86_64 vs 4.18.0-240.22.1.el8_3.x86_64 2021-05-12T08:54:19.512 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-05-12T08:54:19.512 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-05-12T08:54:20.513 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-05-12T08:54:20.514 DEBUG:teuthology.orchestra.run.smithi028:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-05-12T08:54:20.595 INFO:teuthology.orchestra.run.smithi028.stdout:ttyS1 2021-05-12T08:54:20.626 INFO:teuthology.run_tasks:Running task internal.base... 2021-05-12T08:54:20.634 INFO:teuthology.task.internal:Creating test directory... 2021-05-12T08:54:20.635 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-05-12T08:54:20.649 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-05-12T08:54:20.681 INFO:teuthology.run_tasks:Running task internal.archive... 2021-05-12T08:54:20.729 INFO:teuthology.task.internal:Creating archive directory... 2021-05-12T08:54:20.730 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-05-12T08:54:20.754 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-05-12T08:54:20.780 INFO:teuthology.task.internal:Enabling coredump saving... 2021-05-12T08:54:20.781 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-05-12T08:54:20.840 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T08:54:20.852 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T08:54:20.853 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-05-12T08:54:20.864 INFO:teuthology.task.internal:Configuring sudo... 2021-05-12T08:54:20.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-05-12T08:54:20.891 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-05-12T08:54:20.901 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-05-12T08:54:20.902 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-05-12T08:54:20.950 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-05-12T08:54:21.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-05-12T08:54:21.080 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-05-12T08:54:21.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-05-12T08:54:21.210 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2021-05-12T08:54:21.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-05-12T08:54:21.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2021-05-12T08:54:21.387 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-05-12T08:54:21.500 INFO:teuthology.run_tasks:Running task internal.timer... 2021-05-12T08:54:21.509 INFO:teuthology.task.internal:Starting timer... 2021-05-12T08:54:21.509 INFO:teuthology.run_tasks:Running task pcp... 2021-05-12T08:54:21.558 INFO:teuthology.run_tasks:Running task selinux... 2021-05-12T08:54:21.606 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-05-12T08:54:21.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2021-05-12T08:54:21.679 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: [ OK ] 2021-05-12T08:54:21.681 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-05-12T08:54:21.682 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2021-05-12T08:54:21.702 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2021-05-12T08:54:21.703 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive'} 2021-05-12T08:54:21.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-05-12T08:54:21.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-05-12T08:54:21.772 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-05-12T08:54:21.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2021-05-12T08:54:21.842 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-05-12T08:54:21.852 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2021-05-12T08:54:21.859 INFO:teuthology.repo_utils:Fetching master from origin 2021-05-12T08:54:21.890 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-05-12T08:54:21.907 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-05-12T08:54:21.909 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-05-12T08:54:22.257 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-05-12T08:54:23.058 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-05-12T08:54:25.285 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-05-12T08:54:25.308 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:25 +0000 (0:00:00.038) 0:00:00.038 ********* 2021-05-12T08:54:28.159 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi028.front.sepia.ceph.com] 2021-05-12T08:54:28.206 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:28 +0000 (0:00:02.898) 0:00:02.936 ********* 2021-05-12T08:54:28.532 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-05-12T08:54:28.533 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-05-12T08:54:28.555 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:28 +0000 (0:00:00.348) 0:00:03.285 ********* 2021-05-12T08:54:29.131 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-05-12T08:54:29.133 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-05-12T08:54:29.161 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:29 +0000 (0:00:00.606) 0:00:03.891 ********* 2021-05-12T08:54:29.253 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-05-12T08:54:29.255 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:54:29.275 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:29 +0000 (0:00:00.113) 0:00:04.005 ********* 2021-05-12T08:54:29.584 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi028.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T08:54:29.611 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:29 +0000 (0:00:00.335) 0:00:04.341 ********* 2021-05-12T08:54:30.614 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-05-12T08:54:30.616 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1620809669.6714373-4168-92527217358657/source", "state": "file", "uid": 0} 2021-05-12T08:54:30.647 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:30 +0000 (0:00:01.036) 0:00:05.378 ********* 2021-05-12T08:54:31.229 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtC 2021-05-12T08:54:31.231 INFO:teuthology.task.ansible.out:toHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-05-12T08:54:31.293 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-05-12T08:54:31.296 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-05-12T08:54:31.442 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:31 +0000 (0:00:00.795) 0:00:06.173 ********* 2021-05-12T08:54:33.082 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-05-12T08:54:33.084 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] 2021-05-12T08:54:33.136 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-05-12T08:54:33.138 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:33 +0000 (0:00:01.694) 0:00:07.867 ********* 2021-05-12T08:54:33.765 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.161724", "end": "2021-05-12 08:54:33.745208", "rc": 0, "start": "2021-05-12 08:54:33.583484", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} 2021-05-12T08:54:33.786 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:33 +0000 (0:00:00.649) 0:00:08.517 ********* 2021-05-12T08:54:33.989 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-05-12T08:54:33.990 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.010635", "end": "2021-05-12 08:54:33.970827", "failed_when_result": false, "rc": 0, "start": "2021-05-12 08:54:33.960192", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-05-12T08:54:34.012 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:34 +0000 (0:00:00.225) 0:00:08.742 ********* 2021-05-12T08:54:34.080 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-05-12T08:54:34.081 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:54:34.103 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:34 +0000 (0:00:00.091) 0:00:08.833 ********* 2021-05-12T08:54:34.172 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-05-12T08:54:34.173 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:54:34.198 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:34 +0000 (0:00:00.091) 0:00:08.925 ********* 2021-05-12T08:54:34.283 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:54:34.305 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:34 +0000 (0:00:00.110) 0:00:09.036 ********* 2021-05-12T08:54:34.383 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-05-12T08:54:34.385 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:54:34.405 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:34 +0000 (0:00:00.100) 0:00:09.136 ********* 2021-05-12T08:54:34.407 INFO:teuthology.task.ansible.out: 2021-05-12T08:54:34.485 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-05-12T08:54:34.487 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:54:34.508 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:34 +0000 (0:00:00.102) 0:00:09.238 ********* 2021-05-12T08:54:34.752 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-05-12T08:54:34.754 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T08:54:34.774 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:54:34 +0000 (0:00:00.265) 0:00:09.504 ********* 2021-05-12T08:54:35.050 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-05-12T08:54:35.052 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-05-12T08:54:35.261 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-05-12T08:54:35.461 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-05-12T08:54:35.657 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-05-12T08:54:35.855 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-05-12T08:54:36.065 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-05-12T08:54:36.263 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-05-12T08:54:36.462 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-05-12T08:54:36.671 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-05-12T08:54:36.868 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-05-12T08:54:37.072 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-05-12T08:54:37.277 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-05-12T08:54:37.481 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-05-12T08:54:37.683 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-05-12T08:54:37.883 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-05-12T08:54:38.091 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-05-12T08:54:38.290 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-05-12T08:54:38.499 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-05-12T08:54:38.711 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-05-12T08:54:38.909 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-05-12T08:54:39.108 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-05-12T08:54:39.317 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-05-12T08:54:39.540 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-05-12T08:54:39.752 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-05-12T08:54:39.960 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-05-12T08:54:40.157 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-05-12T08:54:40.355 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-05-12T08:54:40.564 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-05-12T08:54:40.762 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-05-12T08:54:40.973 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-05-12T08:54:41.177 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-05-12T08:54:41.373 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-05-12T08:54:41.582 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-05-12T08:54:41.781 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-05-12T08:54:41.979 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-05-12T08:54:42.187 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-05-12T08:54:42.385 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-05-12T08:54:42.584 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-05-12T08:54:42.783 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-05-12T08:54:42.991 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-05-12T08:54:43.192 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-05-12T08:54:43.402 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-05-12T08:54:43.601 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-05-12T08:54:43.801 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-05-12T08:54:44.010 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-05-12T08:54:44.216 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-05-12T08:54:44.418 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-05-12T08:54:44.630 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-05-12T08:54:44.874 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-05-12T08:54:45.087 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-05-12T08:54:45.313 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-05-12T08:54:45.535 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-05-12T08:54:45.736 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-05-12T08:54:45.937 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-05-12T08:54:46.147 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-05-12T08:54:46.355 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-05-12T08:54:46.564 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-05-12T08:54:46.760 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-05-12T08:54:46.969 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-05-12T08:54:47.178 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-05-12T08:54:47.375 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-05-12T08:54:47.582 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-05-12T08:54:47.787 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-05-12T08:54:47.995 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-05-12T08:54:48.195 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-05-12T08:54:48.405 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-05-12T08:54:48.603 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-05-12T08:54:48.802 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-05-12T08:54:49.009 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-05-12T08:54:49.205 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-05-12T08:54:49.412 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-05-12T08:54:49.609 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-05-12T08:54:49.816 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-05-12T08:54:50.014 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-05-12T08:54:50.212 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-05-12T08:54:50.410 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-05-12T08:54:50.620 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-05-12T08:54:50.819 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-05-12T08:54:51.017 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-05-12T08:54:51.229 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-05-12T08:54:51.428 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-05-12T08:54:51.627 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-05-12T08:54:51.840 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-05-12T08:54:52.050 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-05-12T08:54:52.258 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-05-12T08:54:52.458 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-05-12T08:54:52.656 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-05-12T08:54:52.865 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-05-12T08:54:53.073 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-05-12T08:54:53.264 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-05-12T08:54:53.470 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-05-12T08:54:53.667 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-05-12T08:54:53.874 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-05-12T08:54:54.072 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-05-12T08:54:54.279 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-05-12T08:54:54.481 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-05-12T08:54:54.678 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-05-12T08:54:54.875 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-05-12T08:54:55.082 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-05-12T08:54:55.278 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-05-12T08:54:55.479 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-05-12T08:54:55.686 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-05-12T08:54:55.884 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-05-12T08:54:56.083 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-05-12T08:54:56.292 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-05-12T08:54:56.501 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-05-12T08:54:56.701 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-05-12T08:54:56.899 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-05-12T08:54:57.108 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-05-12T08:54:57.305 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-05-12T08:54:57.512 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-05-12T08:54:57.710 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-05-12T08:54:57.917 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-05-12T08:54:58.117 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-05-12T08:54:58.452 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-05-12T08:54:58.660 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-05-12T08:54:58.859 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-05-12T08:54:59.061 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-05-12T08:54:59.269 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-05-12T08:54:59.482 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-05-12T08:54:59.682 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-05-12T08:54:59.893 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-05-12T08:55:00.098 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-05-12T08:55:00.296 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-05-12T08:55:00.499 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-05-12T08:55:00.699 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-05-12T08:55:00.908 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-05-12T08:55:01.106 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-05-12T08:55:01.314 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-05-12T08:55:01.513 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-05-12T08:55:01.713 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-05-12T08:55:01.922 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-05-12T08:55:02.113 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-05-12T08:55:02.324 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-05-12T08:55:02.528 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-05-12T08:55:02.738 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-05-12T08:55:02.937 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-05-12T08:55:03.134 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-05-12T08:55:04.714 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "name": "dachary", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-05-12T08:55:04.835 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:04 +0000 (0:00:30.061) 0:00:39.565 ********* 2021-05-12T08:55:04.892 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-05-12T08:55:04.914 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:04 +0000 (0:00:00.078) 0:00:39.644 ********* 2021-05-12T08:55:05.134 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-05-12T08:55:05.136 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-05-12T08:55:05.138 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-05-12T08:55:05.139 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-05-12T08:55:05.141 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-05-12T08:55:05.143 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de 2021-05-12T08:55:05.144 INFO:teuthology.task.ansible.out:53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d 2021-05-12T08:55:05.146 INFO:teuthology.task.ansible.out:7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846ab 2021-05-12T08:55:05.146 INFO:teuthology.task.ansible.out:d9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumra 2021-05-12T08:55:05.148 INFO:teuthology.task.ansible.out:o", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b76 2021-05-12T08:55:05.149 INFO:teuthology.task.ansible.out:8c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70f 2021-05-12T08:55:05.151 INFO:teuthology.task.ansible.out:f00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41 2021-05-12T08:55:05.153 INFO:teuthology.task.ansible.out:e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b97 2021-05-12T08:55:05.153 INFO:teuthology.task.ansible.out:15003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5 2021-05-12T08:55:05.155 INFO:teuthology.task.ansible.out:dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b 2021-05-12T08:55:05.156 INFO:teuthology.task.ansible.out:1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}]}, "changed": false} 2021-05-12T08:55:05.192 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:05 +0000 (0:00:00.277) 0:00:39.922 ********* 2021-05-12T08:55:06.098 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-05-12T08:55:06.100 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"after": "2f6366fd31821994f3d217ad18c1675bf42a2115", "attempts": 1, "before": "2f6366fd31821994f3d217ad18c1675bf42a2115", "changed": false, "remote_url_changed": false} 2021-05-12T08:55:06.120 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:06 +0000 (0:00:00.928) 0:00:40.851 ********* 2021-05-12T08:55:06.560 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-05-12T08:55:06.561 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-05-12T08:55:06.788 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-05-12T08:55:07.006 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-05-12T08:55:07.223 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-05-12T08:55:07.225 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-05-12T08:55:07.430 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-05-12T08:55:07.647 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-05-12T08:55:07.649 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-05-12T08:55:07.866 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-05-12T08:55:08.073 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-05-12T08:55:08.289 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-05-12T08:55:08.291 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:08.510 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-05-12T08:55:08.512 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-05-12T08:55:08.716 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-05-12T08:55:08.718 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-05-12T08:55:08.727 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:08.738 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:08.945 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-05-12T08:55:09.173 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-05-12T08:55:09.175 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-05-12T08:55:09.380 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-05-12T08:55:09.597 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-05-12T08:55:09.599 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-05-12T08:55:09.805 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-05-12T08:55:10.021 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-05-12T08:55:10.227 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-05-12T08:55:10.447 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-05-12T08:55:10.687 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-05-12T08:55:10.688 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABA 2021-05-12T08:55:10.690 INFO:teuthology.task.ansible.out:AABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-05-12T08:55:10.888 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-05-12T08:55:11.115 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-05-12T08:55:11.332 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-05-12T08:55:11.334 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-05-12T08:55:11.540 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-05-12T08:55:11.750 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-05-12T08:55:11.968 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-05-12T08:55:12.176 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-05-12T08:55:12.397 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-05-12T08:55:12.618 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-05-12T08:55:12.621 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-05-12T08:55:12.839 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-05-12T08:55:13.050 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-05-12T08:55:13.281 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-05-12T08:55:13.493 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-05-12T08:55:13.713 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-05-12T08:55:13.923 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-05-12T08:55:14.142 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-05-12T08:55:14.349 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-05-12T08:55:14.568 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-05-12T08:55:14.570 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-05-12T08:55:14.786 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-05-12T08:55:14.788 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-05-12T08:55:14.993 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-05-12T08:55:15.210 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-05-12T08:55:15.417 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-05-12T08:55:15.634 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtV 2021-05-12T08:55:15.636 INFO:teuthology.task.ansible.out:TFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-05-12T08:55:15.854 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-05-12T08:55:15.858 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-05-12T08:55:15.860 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-05-12T08:55:16.082 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-05-12T08:55:16.310 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-05-12T08:55:16.537 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-05-12T08:55:16.743 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-05-12T08:55:16.959 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-05-12T08:55:17.166 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-05-12T08:55:17.168 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-05-12T08:55:17.382 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-05-12T08:55:17.599 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-05-12T08:55:17.806 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-05-12T08:55:17.807 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-05-12T08:55:18.023 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-05-12T08:55:18.231 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-05-12T08:55:18.449 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-05-12T08:55:18.450 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-05-12T08:55:18.667 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-05-12T08:55:18.887 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-05-12T08:55:19.096 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-05-12T08:55:19.304 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-05-12T08:55:19.523 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-05-12T08:55:19.741 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-05-12T08:55:19.950 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-05-12T08:55:20.182 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-05-12T08:55:20.400 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-05-12T08:55:20.617 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-05-12T08:55:20.824 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-05-12T08:55:21.041 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-05-12T08:55:21.258 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-05-12T08:55:21.475 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-05-12T08:55:21.682 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-05-12T08:55:21.899 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-05-12T08:55:22.115 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-05-12T08:55:22.334 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-05-12T08:55:22.542 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-05-12T08:55:22.761 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-05-12T08:55:22.989 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-05-12T08:55:22.991 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-05-12T08:55:23.195 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-05-12T08:55:23.423 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-05-12T08:55:23.650 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-05-12T08:55:23.857 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-05-12T08:55:24.074 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-05-12T08:55:24.076 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-05-12T08:55:24.291 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-05-12T08:55:24.293 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-05-12T08:55:24.500 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-05-12T08:55:24.723 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-05-12T08:55:24.929 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-05-12T08:55:25.147 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-05-12T08:55:25.355 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-05-12T08:55:25.565 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-05-12T08:55:25.775 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-05-12T08:55:25.993 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-05-12T08:55:25.994 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-05-12T08:55:26.209 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-05-12T08:55:26.416 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-05-12T08:55:26.633 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-05-12T08:55:26.842 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-05-12T08:55:27.060 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-05-12T08:55:27.276 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-05-12T08:55:27.493 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-05-12T08:55:27.495 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-05-12T08:55:27.710 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-05-12T08:55:27.916 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-05-12T08:55:28.133 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-05-12T08:55:28.340 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-05-12T08:55:28.558 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-05-12T08:55:28.784 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-05-12T08:55:28.990 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-05-12T08:55:29.208 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-05-12T08:55:29.425 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-05-12T08:55:29.642 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-05-12T08:55:29.850 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-05-12T08:55:30.067 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-05-12T08:55:30.276 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-05-12T08:55:30.493 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-05-12T08:55:30.700 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-05-12T08:55:30.910 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-05-12T08:55:31.127 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-05-12T08:55:31.334 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-05-12T08:55:31.545 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-05-12T08:55:31.546 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-05-12T08:55:31.763 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-05-12T08:55:31.984 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-05-12T08:55:32.193 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-05-12T08:55:32.411 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-05-12T08:55:32.619 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-05-12T08:55:32.840 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-05-12T08:55:33.046 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-05-12T08:55:33.264 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-05-12T08:55:33.481 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-05-12T08:55:33.688 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-05-12T08:55:33.894 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-05-12T08:55:34.112 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-05-12T08:55:34.318 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-05-12T08:55:34.537 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-05-12T08:55:34.744 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-05-12T08:55:34.957 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-05-12T08:55:35.195 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-05-12T08:55:35.197 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-05-12T08:55:35.403 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-05-12T08:55:35.623 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-05-12T08:55:35.707 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:35 +0000 (0:00:29.586) 0:01:10.437 ********* 2021-05-12T08:55:35.786 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-05-12T08:55:35.788 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:35.797 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:35.799 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:35.809 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:35.811 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:35.820 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:35.831 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:35.832 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:35.843 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.060 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-05-12T08:55:36.071 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.072 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.289 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-05-12T08:55:36.497 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-05-12T08:55:36.498 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.509 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.511 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.520 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.531 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.533 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.542 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.544 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.553 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.555 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.565 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.576 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.577 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.587 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.589 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.598 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.600 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.610 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.611 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.621 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.621 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.632 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.643 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.645 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.655 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.655 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.667 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.669 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.679 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.680 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.690 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.701 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.702 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.713 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.715 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.725 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.726 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.740 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.748 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.750 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.759 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.770 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.772 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.782 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.784 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.795 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.796 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.808 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.809 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.820 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.832 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.832 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.834 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.845 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.856 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.857 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.869 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.871 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.873 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.883 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.885 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.895 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.896 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.908 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.923 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.933 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.934 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.935 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.947 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.948 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.958 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.960 INFO:teuthology.task.ansible.out: skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.971 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.983 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.984 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.995 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:36.997 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.007 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.009 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.018 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.020 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.029 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.040 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.051 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.053 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.062 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.064 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.075 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.085 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.087 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.097 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.098 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.108 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.110 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.119 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.121 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.130 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.141 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.143 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.153 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.155 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.164 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.166 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.175 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.186 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.188 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.197 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.199 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.208 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.210 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.220 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.230 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.232 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.242 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.243 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.253 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.264 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.265 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.275 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.277 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.287 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.289 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.336 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:37 +0000 (0:00:01.628) 0:01:12.066 ********* 2021-05-12T08:55:37.404 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:37.425 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:37 +0000 (0:00:00.089) 0:01:12.155 ********* 2021-05-12T08:55:37.722 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi028.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-05-12T08:55:37.910 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-05-12T08:55:38.148 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-05-12T08:55:38.342 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-05-12T08:55:38.551 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-05-12T08:55:38.739 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-05-12T08:55:38.937 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-05-12T08:55:39.145 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-05-12T08:55:39.343 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-05-12T08:55:39.540 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-05-12T08:55:39.736 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-05-12T08:55:39.943 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-05-12T08:55:40.141 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-05-12T08:55:40.337 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-05-12T08:55:40.533 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-05-12T08:55:40.731 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-05-12T08:55:40.923 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-05-12T08:55:41.119 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-05-12T08:55:41.318 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-05-12T08:55:41.514 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-05-12T08:55:41.722 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-05-12T08:55:41.919 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-05-12T08:55:42.116 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-05-12T08:55:42.312 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-05-12T08:55:42.508 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-05-12T08:55:42.715 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-05-12T08:55:42.916 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-05-12T08:55:43.113 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-05-12T08:55:43.319 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-05-12T08:55:43.516 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-05-12T08:55:43.712 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-05-12T08:55:43.909 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-05-12T08:55:44.104 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-05-12T08:55:44.300 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-05-12T08:55:44.498 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-05-12T08:55:44.704 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-05-12T08:55:44.897 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-05-12T08:55:45.094 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-05-12T08:55:46.834 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jdillaman", "name": "jdillaman", "remove": false, "state": "absent"} 2021-05-12T08:55:47.047 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-05-12T08:55:47.253 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-05-12T08:55:47.462 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-05-12T08:55:47.658 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-05-12T08:55:47.856 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-05-12T08:55:48.052 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-05-12T08:55:48.252 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-05-12T08:55:48.458 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-05-12T08:55:48.654 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-05-12T08:55:48.852 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-05-12T08:55:48.904 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:48 +0000 (0:00:11.478) 0:01:23.634 ********* 2021-05-12T08:55:49.453 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-05-12T08:55:49.455 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "checksum": "313dfb187fdfe6062591abb65b175f610cf74a8a", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "b87ac93b406a130b21ee0476c1efad05", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1620809749.001566-9627-67928250311176/source", "state": "file", "uid": 0} 2021-05-12T08:55:49.476 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:49 +0000 (0:00:00.572) 0:01:24.207 ********* 2021-05-12T08:55:49.545 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-05-12T08:55:49.546 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => { "msg": "Host smithi028.front.sepia.ceph.com is running CentOS 8.3 (n/a)" } 2021-05-12T08:55:49.569 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:49 +0000 (0:00:00.090) 0:01:24.298 ********* 2021-05-12T08:55:49.636 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-05-12T08:55:49.637 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-05-12T08:55:49.659 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:49 +0000 (0:00:00.091) 0:01:24.389 ********* 2021-05-12T08:55:49.727 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-05-12T08:55:49.729 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:49.749 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:49 +0000 (0:00:00.090) 0:01:24.479 ********* 2021-05-12T08:55:50.291 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-05-12T08:55:50.293 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.312355", "end": "2021-05-12 08:55:50.272671", "rc": 0, "start": "2021-05-12 08:55:49.960316", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-05-12T08:55:50.314 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:50 +0000 (0:00:00.565) 0:01:25.045 ********* 2021-05-12T08:55:50.404 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:50.426 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:50 +0000 (0:00:00.111) 0:01:25.156 ********* 2021-05-12T08:55:50.495 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-05-12T08:55:50.497 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:50.517 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:50 +0000 (0:00:00.091) 0:01:25.248 ********* 2021-05-12T08:55:50.595 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:50.618 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:50 +0000 (0:00:00.100) 0:01:25.348 ********* 2021-05-12T08:55:50.707 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:50.729 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:50 +0000 (0:00:00.110) 0:01:25.459 ********* 2021-05-12T08:55:50.797 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-05-12T08:55:50.799 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:50.821 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:50 +0000 (0:00:00.092) 0:01:25.552 ********* 2021-05-12T08:55:50.890 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-05-12T08:55:50.891 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:50.912 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:50 +0000 (0:00:00.090) 0:01:25.642 ********* 2021-05-12T08:55:50.993 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-05-12T08:55:50.994 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi028.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-12T08:55:51.017 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:51 +0000 (0:00:00.105) 0:01:25.747 ********* 2021-05-12T08:55:51.086 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-05-12T08:55:51.088 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:51.109 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:51 +0000 (0:00:00.092) 0:01:25.840 ********* 2021-05-12T08:55:51.178 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-05-12T08:55:51.181 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:51.200 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:51 +0000 (0:00:00.090) 0:01:25.930 ********* 2021-05-12T08:55:51.279 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-05-12T08:55:51.281 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:51.301 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:51 +0000 (0:00:00.100) 0:01:26.031 ********* 2021-05-12T08:55:51.380 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-05-12T08:55:51.382 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-12T08:55:51.403 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:51 +0000 (0:00:00.102) 0:01:26.133 ********* 2021-05-12T08:55:51.472 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-05-12T08:55:51.474 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:51.496 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:51 +0000 (0:00:00.093) 0:01:26.226 ********* 2021-05-12T08:55:51.567 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi028.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-12T08:55:51.590 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:51 +0000 (0:00:00.093) 0:01:26.320 ********* 2021-05-12T08:55:51.667 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-05-12T08:55:51.669 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:51.691 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:51 +0000 (0:00:00.100) 0:01:26.421 ********* 2021-05-12T08:55:51.760 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-05-12T08:55:51.762 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:51.783 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:51 +0000 (0:00:00.092) 0:01:26.514 ********* 2021-05-12T08:55:51.862 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-05-12T08:55:51.864 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:51.887 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:51 +0000 (0:00:00.103) 0:01:26.617 ********* 2021-05-12T08:55:51.956 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-05-12T08:55:51.958 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:51.984 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:51 +0000 (0:00:00.097) 0:01:26.714 ********* 2021-05-12T08:55:52.052 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-05-12T08:55:52.054 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-12T08:55:52.085 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:52 +0000 (0:00:00.101) 0:01:26.815 ********* 2021-05-12T08:55:52.156 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-05-12T08:55:52.159 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:52.189 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:52 +0000 (0:00:00.103) 0:01:26.919 ********* 2021-05-12T08:55:52.259 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-05-12T08:55:52.261 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:52.283 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:52 +0000 (0:00:00.094) 0:01:27.013 ********* 2021-05-12T08:55:52.350 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-05-12T08:55:52.351 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:52.371 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:52 +0000 (0:00:00.088) 0:01:27.102 ********* 2021-05-12T08:55:52.439 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-05-12T08:55:52.441 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:52.461 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:52 +0000 (0:00:00.089) 0:01:27.191 ********* 2021-05-12T08:55:52.539 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:52.560 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:52 +0000 (0:00:00.099) 0:01:27.291 ********* 2021-05-12T08:55:52.638 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-05-12T08:55:52.640 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:52.664 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:52 +0000 (0:00:00.104) 0:01:27.395 ********* 2021-05-12T08:55:52.731 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-05-12T08:55:52.733 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-05-12T08:55:52.753 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:52 +0000 (0:00:00.088) 0:01:27.483 ********* 2021-05-12T08:55:52.820 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:52.842 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:52 +0000 (0:00:00.089) 0:01:27.572 ********* 2021-05-12T08:55:52.910 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-05-12T08:55:52.911 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:52.931 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:52 +0000 (0:00:00.089) 0:01:27.662 ********* 2021-05-12T08:55:53.207 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-05-12T08:55:53.209 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T08:55:53.229 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:53 +0000 (0:00:00.297) 0:01:27.959 ********* 2021-05-12T08:55:54.035 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-05-12T08:55:54.036 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-05-12T08:55:54.472 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-05-12T08:55:54.493 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:54 +0000 (0:00:01.264) 0:01:29.224 ********* 2021-05-12T08:55:55.019 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-05-12T08:55:55.021 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-05-12T08:55:55.465 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-05-12T08:55:55.488 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:55 +0000 (0:00:00.994) 0:01:30.218 ********* 2021-05-12T08:55:55.565 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-05-12T08:55:55.567 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:55.587 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:55 +0000 (0:00:00.099) 0:01:30.318 ********* 2021-05-12T08:55:55.654 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-05-12T08:55:55.656 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:55.676 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:55 +0000 (0:00:00.088) 0:01:30.406 ********* 2021-05-12T08:55:55.743 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-05-12T08:55:55.744 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:55.764 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:55 +0000 (0:00:00.088) 0:01:30.495 ********* 2021-05-12T08:55:55.832 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-05-12T08:55:55.834 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:55.854 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:55 +0000 (0:00:00.089) 0:01:30.584 ********* 2021-05-12T08:55:55.921 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-05-12T08:55:55.923 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-05-12T08:55:55.932 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-05-12T08:55:55.934 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi028.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-05-12T08:55:55.956 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:55 +0000 (0:00:00.102) 0:01:30.687 ********* 2021-05-12T08:55:56.013 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-05-12T08:55:56.015 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:56.035 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:56 +0000 (0:00:00.078) 0:01:30.765 ********* 2021-05-12T08:55:56.093 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:56.114 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:56 +0000 (0:00:00.079) 0:01:30.845 ********* 2021-05-12T08:55:56.183 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-05-12T08:55:56.185 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:56.205 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:56 +0000 (0:00:00.090) 0:01:30.936 ********* 2021-05-12T08:55:56.263 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-05-12T08:55:56.264 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:56.284 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:56 +0000 (0:00:00.079) 0:01:31.015 ********* 2021-05-12T08:55:56.352 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-05-12T08:55:56.354 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:55:56.374 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:56 +0000 (0:00:00.088) 0:01:31.104 ********* 2021-05-12T08:55:57.520 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.953095", "end": "2021-05-12 08:55:57.500350", "rc": 0, "start": "2021-05-12 08:55:56.547255", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T08:55:57.541 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:55:57 +0000 (0:00:01.167) 0:01:32.272 ********* 2021-05-12T08:56:01.682 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-05-12T08:56:01.684 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T08:56:01.703 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:01 +0000 (0:00:04.162) 0:01:36.434 ********* 2021-05-12T08:56:01.771 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-05-12T08:56:01.773 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:56:01.793 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:01 +0000 (0:00:00.089) 0:01:36.523 ********* 2021-05-12T08:56:01.860 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-05-12T08:56:01.862 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:56:01.890 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:01 +0000 (0:00:00.097) 0:01:36.621 ********* 2021-05-12T08:56:01.960 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-05-12T08:56:01.961 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:56:01.982 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:01 +0000 (0:00:00.091) 0:01:36.712 ********* 2021-05-12T08:56:02.049 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-05-12T08:56:02.051 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:56:02.071 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:02 +0000 (0:00:00.089) 0:01:36.802 ********* 2021-05-12T08:56:02.565 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-05-12T08:56:02.567 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-05-12T08:56:02.587 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:02 +0000 (0:00:00.515) 0:01:37.317 ********* 2021-05-12T08:56:04.031 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-05-12T08:56:04.033 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T08:56:04.052 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:04 +0000 (0:00:01.465) 0:01:38.783 ********* 2021-05-12T08:56:04.586 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-05-12T08:56:04.588 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-05-12T08:56:05.032 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-05-12T08:56:05.479 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-05-12T08:56:05.502 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:05 +0000 (0:00:01.449) 0:01:40.233 ********* 2021-05-12T08:56:05.695 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-05-12T08:56:05.697 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-12T08:56:05.719 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:05 +0000 (0:00:00.217) 0:01:40.450 ********* 2021-05-12T08:56:06.213 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-05-12T08:56:06.215 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-05-12T08:56:06.628 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-05-12T08:56:07.057 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-05-12T08:56:07.081 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:07 +0000 (0:00:01.361) 0:01:41.811 ********* 2021-05-12T08:56:07.149 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-05-12T08:56:07.151 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:56:07.170 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:07 +0000 (0:00:00.089) 0:01:41.901 ********* 2021-05-12T08:56:07.238 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-05-12T08:56:07.240 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-05-12T08:56:07.262 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:07 +0000 (0:00:00.091) 0:01:41.992 ********* 2021-05-12T08:56:07.496 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-05-12T08:56:07.498 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-05-12T08:56:07.518 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:07 +0000 (0:00:00.256) 0:01:42.249 ********* 2021-05-12T08:56:07.596 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-05-12T08:56:07.598 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-12T08:56:07.607 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-12T08:56:07.618 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-12T08:56:07.620 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-05-12T08:56:07.641 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:07 +0000 (0:00:00.122) 0:01:42.371 ********* 2021-05-12T08:56:13.097 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-05-12T08:56:13.098 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: nagios-plugins-2.3.3-5.el8.x86_64", "Installed: nagios-plugins-load-2.3.3-5.el8.x86_64", "Installed: nrpe-4.0.3-6.el8.x86_64", "Installed: nagios-common-4.4.6-4.el8.x86_64", "Removed: nagios-common-4.4.5-1.el8.x86_64", "Removed: nagios-plugins-2.3.3-4.el8.x86_64", "Removed: nagios-plugins-load-2.3.3-4.el8.x86_64", "Removed: nrpe-4.0.3-1.el8.x86_64"]} 2021-05-12T08:56:13.122 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:13 +0000 (0:00:05.481) 0:01:47.853 ********* 2021-05-12T08:56:13.201 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-05-12T08:56:13.203 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:56:13.224 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:13 +0000 (0:00:00.102) 0:01:47.955 ********* 2021-05-12T08:56:13.737 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-05-12T08:56:13.739 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-05-12T08:56:13.759 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:13 +0000 (0:00:00.535) 0:01:48.490 ********* 2021-05-12T08:56:14.264 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-05-12T08:56:14.266 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-05-12T08:56:14.288 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:14 +0000 (0:00:00.528) 0:01:49.018 ********* 2021-05-12T08:56:14.356 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-05-12T08:56:14.358 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-05-12T08:56:14.367 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-05-12T08:56:14.395 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:14 +0000 (0:00:00.107) 0:01:49.126 ********* 2021-05-12T08:56:14.641 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-05-12T08:56:14.643 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T08:56:14.664 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:14 +0000 (0:00:00.267) 0:01:49.394 ********* 2021-05-12T08:56:14.950 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-05-12T08:56:14.952 INFO:teuthology.task.ansible.out:fatal: [smithi028.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-05-12T08:56:14.976 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:14 +0000 (0:00:00.312) 0:01:49.706 ********* 2021-05-12T08:56:15.047 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-05-12T08:56:15.049 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:56:15.071 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:15 +0000 (0:00:00.095) 0:01:49.802 ********* 2021-05-12T08:56:15.595 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-05-12T08:56:15.597 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-05-12T08:56:15.617 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:15 +0000 (0:00:00.546) 0:01:50.348 ********* 2021-05-12T08:56:16.362 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-05-12T08:56:16.364 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-05-12 08:56:11 UTC", "ActiveEnterTimestampMonotonic": "136966882", "ActiveExitTimestamp": "Wed 2021-05-12 08:56:11 UTC", "ActiveExitTimestampMonotonic": "136946305", "ActiveState": "active", "After": "network-online.target var-run.mount system.slice -.mount sysinit.target local-fs.target nss-lookup.target tmp.mount systemd-journald.socket time-sync.target systemd-tmpfiles-setup.service basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-05-12 08:56:11 UTC", "AssertTimestampMonotonic": "136966093", "Before": "multi-user.target shutdown.target getty@tty1.service xdm.service plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-05-12 08:56:11 UTC", "Co 2021-05-12T08:56:16.365 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "136966093", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "14912", "ExecMainStartTimestamp": "Wed 2021-05-12 08:56:11 UTC", "ExecMainStartTimestampMonotonic": "136966857", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "thawing", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Wed 2021-05-12 08:56:11 UTC", "InactiveEnterTimestampMonotonic": "136965374", "InactiveExitTimestamp": "Wed 2021-05-12 08:56:11 UTC", 2021-05-12T08:56:16.367 INFO:teuthology.task.ansible.out:"InactiveExitTimestampMonotonic": "136966882", "InvocationID": "5a67292e87b54510b803a085c1125d4e", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127277", "LimitNPROCSoft": "127277", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127277", "LimitSIGPENDINGSoft": "127277", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "14912", "MemoryAccounting": "yes", "MemoryCurrent": "876544", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateT 2021-05-12T08:56:16.368 INFO:teuthology.task.ansible.out:mp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-05-12 08:56:11 UTC", "StateChangeTimestampMonotonic": "136966882", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203643", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-05-12 08:56:11 UTC", "WatchdogTimestampMonotonic": "136966882", "WatchdogUSec": "0"}} 2021-05-12T08:56:16.385 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:16 +0000 (0:00:00.768) 0:01:51.116 ********* 2021-05-12T08:56:16.609 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-05-12T08:56:16.611 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001783", "end": "2021-05-12 08:56:16.594010", "rc": 0, "start": "2021-05-12 08:56:16.592227", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-05-12T08:56:16.631 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:16 +0000 (0:00:00.245) 0:01:51.361 ********* 2021-05-12T08:56:18.068 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-05-12T08:56:18.070 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T08:56:18.091 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:18 +0000 (0:00:01.460) 0:01:52.821 ********* 2021-05-12T08:56:57.770 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-05-12T08:56:57.772 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-54.el8_3.4.noarch", "Installed: selinux-policy-targeted-3.14.3-54.el8_3.4.noarch", "Removed: selinux-policy-3.14.3-54.el8_3.2.noarch", "Removed: selinux-policy-targeted-3.14.3-54.el8_3.2.noarch"]} 2021-05-12T08:56:57.793 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:57 +0000 (0:00:39.701) 0:02:32.523 ********* 2021-05-12T08:56:58.598 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-05-12T08:56:58.620 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:56:58 +0000 (0:00:00.826) 0:02:33.350 ********* 2021-05-12T08:57:15.824 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-05-12T08:57:15.825 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.964814", "end": "2021-05-12 08:57:15.795709", "failed_when_result": false, "rc": 0, "start": "2021-05-12 08:56:58.830895", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-05-12T08:57:15.847 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:15 +0000 (0:00:17.227) 0:02:50.578 ********* 2021-05-12T08:57:16.343 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-05-12T08:57:16.365 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:16 +0000 (0:00:00.517) 0:02:51.095 ********* 2021-05-12T08:57:16.613 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-05-12T08:57:16.616 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.015202", "end": "2021-05-12 08:57:16.587688", "rc": 0, "start": "2021-05-12 08:57:16.572486", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T08:57:16.640 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:16 +0000 (0:00:00.275) 0:02:51.370 ********* 2021-05-12T08:57:16.885 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-05-12T08:57:16.887 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.011501", "end": "2021-05-12 08:57:16.863781", "rc": 0, "start": "2021-05-12 08:57:16.852280", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T08:57:16.908 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:16 +0000 (0:00:00.268) 0:02:51.639 ********* 2021-05-12T08:57:34.141 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-05-12T08:57:34.143 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.993692", "end": "2021-05-12 08:57:34.115579", "rc": 0, "start": "2021-05-12 08:57:17.121887", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T08:57:34.163 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:34 +0000 (0:00:17.255) 0:03:08.894 ********* 2021-05-12T08:57:34.398 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-05-12T08:57:34.420 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:34 +0000 (0:00:00.257) 0:03:09.151 ********* 2021-05-12T08:57:34.488 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-05-12T08:57:34.490 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-05-12T08:57:34.510 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:34 +0000 (0:00:00.089) 0:03:09.241 ********* 2021-05-12T08:57:34.578 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-05-12T08:57:34.580 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-05-12T08:57:34.611 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:34 +0000 (0:00:00.101) 0:03:09.342 ********* 2021-05-12T08:57:34.690 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-05-12T08:57:34.692 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-05-12T08:57:34.713 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:34 +0000 (0:00:00.101) 0:03:09.444 ********* 2021-05-12T08:57:34.784 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi028.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-05-12T08:57:34.805 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:34 +0000 (0:00:00.091) 0:03:09.535 ********* 2021-05-12T08:57:34.997 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-05-12T08:57:34.998 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-05-12T08:57:35.018 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:35 +0000 (0:00:00.213) 0:03:09.749 ********* 2021-05-12T08:57:35.252 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-05-12T08:57:35.254 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-05-12T08:57:35.275 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:35 +0000 (0:00:00.257) 0:03:10.006 ********* 2021-05-12T08:57:35.333 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-05-12T08:57:35.335 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:57:35.355 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:35 +0000 (0:00:00.079) 0:03:10.085 ********* 2021-05-12T08:57:35.412 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-05-12T08:57:35.414 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:57:35.436 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:35 +0000 (0:00:00.081) 0:03:10.167 ********* 2021-05-12T08:57:35.692 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi028.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-05-12T08:57:35.716 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:35 +0000 (0:00:00.279) 0:03:10.446 ********* 2021-05-12T08:57:36.307 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-05-12T08:57:36.309 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-05-12T08:57:36.329 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:36 +0000 (0:00:00.613) 0:03:11.059 ********* 2021-05-12T08:57:36.397 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-05-12T08:57:36.399 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi028"}, "changed": false} 2021-05-12T08:57:36.419 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:36 +0000 (0:00:00.089) 0:03:11.149 ********* 2021-05-12T08:57:37.474 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-05-12T08:57:37.475 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi028.front.sepia.ceph.com", "ansible_hostname": "smithi028", "ansible_nodename": "smithi028"}, "changed": true, "name": "smithi028"} 2021-05-12T08:57:37.495 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:37 +0000 (0:00:01.076) 0:03:12.226 ********* 2021-05-12T08:57:37.987 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-05-12T08:57:37.989 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-05-12T08:57:38.008 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:38 +0000 (0:00:00.513) 0:03:12.739 ********* 2021-05-12T08:57:38.517 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-05-12T08:57:38.537 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:38 +0000 (0:00:00.528) 0:03:13.268 ********* 2021-05-12T08:57:39.152 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-05-12T08:57:39.153 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-05-12T08:57:39.182 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:39 +0000 (0:00:00.644) 0:03:13.913 ********* 2021-05-12T08:57:39.697 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-05-12T08:57:39.723 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:39 +0000 (0:00:00.540) 0:03:14.454 ********* 2021-05-12T08:57:39.957 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-05-12T08:57:39.959 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-05-12T08:57:39.980 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:39 +0000 (0:00:00.256) 0:03:14.710 ********* 2021-05-12T08:57:40.235 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-05-12T08:57:40.256 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:40 +0000 (0:00:00.276) 0:03:14.986 ********* 2021-05-12T08:57:40.547 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-05-12T08:57:40.549 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-05-12T08:57:40.569 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:40 +0000 (0:00:00.313) 0:03:15.300 ********* 2021-05-12T08:57:41.096 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-05-12T08:57:41.098 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-05-12T08:57:41.117 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:41 +0000 (0:00:00.548) 0:03:15.848 ********* 2021-05-12T08:57:41.643 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-05-12T08:57:41.645 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-05-12T08:57:41.666 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:41 +0000 (0:00:00.547) 0:03:16.396 ********* 2021-05-12T08:57:44.195 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-05-12T08:57:44.197 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.279542", "end": "2021-05-12 08:57:44.171569", "rc": 0, "start": "2021-05-12 08:57:41.892027", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T08:57:44.218 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:44 +0000 (0:00:02.552) 0:03:18.948 ********* 2021-05-12T08:57:44.745 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-05-12T08:57:44.747 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-05-12T08:57:45.208 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-05-12T08:57:45.648 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-05-12T08:57:45.671 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:45 +0000 (0:00:01.453) 0:03:20.402 ********* 2021-05-12T08:57:46.187 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-05-12T08:57:46.189 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-05-12T08:57:46.212 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:46 +0000 (0:00:00.540) 0:03:20.942 ********* 2021-05-12T08:57:46.747 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-05-12T08:57:46.749 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-05-12T08:57:47.184 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-05-12T08:57:47.612 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-05-12T08:57:47.637 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:47 +0000 (0:00:01.425) 0:03:22.367 ********* 2021-05-12T08:57:47.705 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-05-12T08:57:47.728 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:47 +0000 (0:00:00.090) 0:03:22.458 ********* 2021-05-12T08:57:47.797 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-05-12T08:57:47.799 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:57:47.819 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:47 +0000 (0:00:00.091) 0:03:22.549 ********* 2021-05-12T08:57:47.887 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-05-12T08:57:47.889 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-05-12T08:57:47.899 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-05-12T08:57:47.921 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:47 +0000 (0:00:00.102) 0:03:22.651 ********* 2021-05-12T08:57:47.990 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-05-12T08:57:47.991 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:57:48.012 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:48 +0000 (0:00:00.090) 0:03:22.742 ********* 2021-05-12T08:57:48.081 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T08:57:48.082 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-05-12T08:57:48.103 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:48 +0000 (0:00:00.090) 0:03:22.833 ********* 2021-05-12T08:57:48.170 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-05-12T08:57:48.171 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:57:48.191 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:48 +0000 (0:00:00.088) 0:03:22.922 ********* 2021-05-12T08:57:48.259 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-05-12T08:57:48.261 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:57:48.281 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:48 +0000 (0:00:00.089) 0:03:23.011 ********* 2021-05-12T08:57:48.526 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-05-12T08:57:48.528 INFO:teuthology.task.ansible.out:fatal: [smithi028.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007460", "end": "2021-05-12 08:57:48.506004", "msg": "non-zero return code", "rc": 1, "start": "2021-05-12 08:57:48.498544", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-05-12T08:57:48.548 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:48 +0000 (0:00:00.267) 0:03:23.279 ********* 2021-05-12T08:57:48.637 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:57:48.658 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:48 +0000 (0:00:00.109) 0:03:23.389 ********* 2021-05-12T08:57:51.630 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-05-12T08:57:51.631 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64"]} 2021-05-12T08:57:51.652 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:51 +0000 (0:00:02.994) 0:03:26.383 ********* 2021-05-12T08:57:54.326 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-05-12T08:57:54.348 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:57:54 +0000 (0:00:02.695) 0:03:29.078 ********* 2021-05-12T08:58:01.301 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-05-12T08:58:01.302 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-common-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-core-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:4.2.0-34.module_el8.3.0+755+88436ea4.5.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: qemu-kvm-common-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: qemu-kvm-core-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64"]} 2021-05-12T08:58:01.323 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:01 +0000 (0:00:06.975) 0:03:36.053 ********* 2021-05-12T08:58:01.390 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-05-12T08:58:01.392 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:01.412 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:01 +0000 (0:00:00.089) 0:03:36.143 ********* 2021-05-12T08:58:02.943 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T08:58:02.965 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:02 +0000 (0:00:01.552) 0:03:37.695 ********* 2021-05-12T08:58:04.402 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-05-12T08:58:04.403 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T08:58:04.423 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:04 +0000 (0:00:01.458) 0:03:39.154 ********* 2021-05-12T08:58:04.772 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-05-12T08:58:04.773 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service sysinit.target basic.target dbus.socket system.slice polkit.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target iptables.service ip6tables.service ipset.service ebtables.service nftables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "D 2021-05-12T08:58:04.775 INFO:teuthology.task.ansible.out:evicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "Limi 2021-05-12T08:58:04.776 INFO:teuthology.task.ansible.out:tNOFILESoft": "1024", "LimitNPROC": "127277", "LimitNPROCSoft": "127277", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127277", "LimitSIGPENDINGSoft": "127277", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket system.slice sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "S 2021-05-12T08:58:04.778 INFO:teuthology.task.ansible.out:tartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203643", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-05-12T08:58:04.796 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:04 +0000 (0:00:00.372) 0:03:39.527 ********* 2021-05-12T08:58:05.168 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-05-12T08:58:05.170 INFO:teuthology.task.ansible.out:fatal: [smithi028.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-05-12T08:58:05.189 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:05 +0000 (0:00:00.393) 0:03:39.920 ********* 2021-05-12T08:58:05.892 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-05-12T08:58:05.893 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-05-12T08:58:05.914 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:05 +0000 (0:00:00.724) 0:03:40.644 ********* 2021-05-12T08:58:05.982 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-05-12T08:58:05.982 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:06.004 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:05 +0000 (0:00:00.089) 0:03:40.734 ********* 2021-05-12T08:58:06.074 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-05-12T08:58:06.075 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:06.096 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:06 +0000 (0:00:00.092) 0:03:40.826 ********* 2021-05-12T08:58:06.163 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-05-12T08:58:06.165 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:06.185 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:06 +0000 (0:00:00.088) 0:03:40.915 ********* 2021-05-12T08:58:06.252 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-05-12T08:58:06.254 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:06.273 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:06 +0000 (0:00:00.088) 0:03:41.004 ********* 2021-05-12T08:58:06.341 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:06.363 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:06 +0000 (0:00:00.089) 0:03:41.093 ********* 2021-05-12T08:58:06.431 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-05-12T08:58:06.468 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 12 May 2021 08:58:06 +0000 (0:00:00.090) 0:03:41.184 ********* 2021-05-12T08:58:06.543 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-05-12T08:58:06.564 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:06 +0000 (0:00:00.110) 0:03:41.294 ********* 2021-05-12T08:58:06.642 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-05-12T08:58:06.644 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:06.664 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:06 +0000 (0:00:00.100) 0:03:41.395 ********* 2021-05-12T08:58:06.732 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-05-12T08:58:06.734 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:06.757 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:06 +0000 (0:00:00.092) 0:03:41.487 ********* 2021-05-12T08:58:06.826 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-05-12T08:58:06.828 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-05-12T08:58:06.837 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-05-12T08:58:06.860 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:06 +0000 (0:00:00.102) 0:03:41.590 ********* 2021-05-12T08:58:06.917 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-05-12T08:58:06.919 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:06.940 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:06 +0000 (0:00:00.080) 0:03:41.670 ********* 2021-05-12T08:58:06.998 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-05-12T08:58:07.021 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:07 +0000 (0:00:00.080) 0:03:41.751 ********* 2021-05-12T08:58:07.088 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-05-12T08:58:07.090 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:07.110 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:07 +0000 (0:00:00.089) 0:03:41.840 ********* 2021-05-12T08:58:07.168 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-05-12T08:58:07.169 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:07.189 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:07 +0000 (0:00:00.079) 0:03:41.920 ********* 2021-05-12T08:58:07.278 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-05-12T08:58:07.280 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:07.299 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:07 +0000 (0:00:00.109) 0:03:42.030 ********* 2021-05-12T08:58:07.368 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-05-12T08:58:07.369 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:07.390 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:07 +0000 (0:00:00.090) 0:03:42.120 ********* 2021-05-12T08:58:07.457 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-05-12T08:58:07.459 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:07.479 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:07 +0000 (0:00:00.089) 0:03:42.210 ********* 2021-05-12T08:58:07.547 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-05-12T08:58:07.549 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:07.570 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:07 +0000 (0:00:00.089) 0:03:42.300 ********* 2021-05-12T08:58:07.637 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-05-12T08:58:07.639 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:07.659 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:07 +0000 (0:00:00.089) 0:03:42.390 ********* 2021-05-12T08:58:07.727 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-05-12T08:58:07.729 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:07.749 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:07 +0000 (0:00:00.089) 0:03:42.480 ********* 2021-05-12T08:58:07.817 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-05-12T08:58:07.819 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:07.842 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:07 +0000 (0:00:00.092) 0:03:42.572 ********* 2021-05-12T08:58:07.909 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-05-12T08:58:07.911 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:07.931 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:07 +0000 (0:00:00.089) 0:03:42.661 ********* 2021-05-12T08:58:08.009 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-05-12T08:58:08.010 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:08.031 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:08 +0000 (0:00:00.099) 0:03:42.761 ********* 2021-05-12T08:58:08.100 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-05-12T08:58:08.102 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:08.122 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:08 +0000 (0:00:00.091) 0:03:42.853 ********* 2021-05-12T08:58:08.201 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-05-12T08:58:08.202 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:08.222 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:08 +0000 (0:00:00.100) 0:03:42.953 ********* 2021-05-12T08:58:08.280 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-05-12T08:58:08.282 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:08.303 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:08 +0000 (0:00:00.079) 0:03:43.033 ********* 2021-05-12T08:58:08.370 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-05-12T08:58:08.372 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:08.392 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:08 +0000 (0:00:00.089) 0:03:43.122 ********* 2021-05-12T08:58:08.472 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:08.493 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:08 +0000 (0:00:00.100) 0:03:43.223 ********* 2021-05-12T08:58:08.561 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-05-12T08:58:08.563 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:08.583 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:08 +0000 (0:00:00.089) 0:03:43.313 ********* 2021-05-12T08:58:08.653 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:08.676 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:08 +0000 (0:00:00.092) 0:03:43.406 ********* 2021-05-12T08:58:08.745 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-05-12T08:58:08.746 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:08.767 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:08 +0000 (0:00:00.090) 0:03:43.497 ********* 2021-05-12T08:58:08.837 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-05-12T08:58:08.839 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:08.859 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:08 +0000 (0:00:00.092) 0:03:43.589 ********* 2021-05-12T08:58:08.926 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-05-12T08:58:08.928 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:08.951 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:08 +0000 (0:00:00.091) 0:03:43.681 ********* 2021-05-12T08:58:09.009 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-05-12T08:58:09.010 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:09.031 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:09 +0000 (0:00:00.079) 0:03:43.761 ********* 2021-05-12T08:58:09.088 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-05-12T08:58:09.090 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:09.111 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:09 +0000 (0:00:00.080) 0:03:43.841 ********* 2021-05-12T08:58:09.169 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T08:58:09.170 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:09.191 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:09 +0000 (0:00:00.079) 0:03:43.921 ********* 2021-05-12T08:58:09.248 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-05-12T08:58:09.250 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:09.271 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:09 +0000 (0:00:00.079) 0:03:44.001 ********* 2021-05-12T08:58:09.327 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-05-12T08:58:09.329 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:09.349 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:09 +0000 (0:00:00.078) 0:03:44.079 ********* 2021-05-12T08:58:09.416 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-05-12T08:58:09.417 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:09.437 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:09 +0000 (0:00:00.088) 0:03:44.168 ********* 2021-05-12T08:58:09.504 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-05-12T08:58:09.506 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:09.526 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:09 +0000 (0:00:00.088) 0:03:44.256 ********* 2021-05-12T08:58:09.778 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-05-12T08:58:09.781 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-05-12T08:58:09.800 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:09 +0000 (0:00:00.274) 0:03:44.531 ********* 2021-05-12T08:58:10.056 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-05-12T08:58:10.057 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-12T08:58:10.078 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:10 +0000 (0:00:00.277) 0:03:44.808 ********* 2021-05-12T08:58:10.322 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-05-12T08:58:10.324 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-05-12T08:58:10.344 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:10 +0000 (0:00:00.266) 0:03:45.075 ********* 2021-05-12T08:58:10.412 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-05-12T08:58:10.414 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:10.436 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:10 +0000 (0:00:00.092) 0:03:45.167 ********* 2021-05-12T08:58:10.504 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-05-12T08:58:10.506 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:10.526 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:10 +0000 (0:00:00.089) 0:03:45.256 ********* 2021-05-12T08:58:10.594 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-05-12T08:58:10.596 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:10.616 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:10 +0000 (0:00:00.089) 0:03:45.346 ********* 2021-05-12T08:58:10.673 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-05-12T08:58:10.675 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:10.695 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:10 +0000 (0:00:00.078) 0:03:45.425 ********* 2021-05-12T08:58:10.762 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:10.783 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:10 +0000 (0:00:00.088) 0:03:45.514 ********* 2021-05-12T08:58:10.841 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-05-12T08:58:10.843 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:10.873 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:10 +0000 (0:00:00.089) 0:03:45.604 ********* 2021-05-12T08:58:10.944 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:10.965 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:10 +0000 (0:00:00.092) 0:03:45.696 ********* 2021-05-12T08:58:11.033 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-05-12T08:58:11.035 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:11.055 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:11 +0000 (0:00:00.089) 0:03:45.786 ********* 2021-05-12T08:58:11.123 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-05-12T08:58:11.125 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:11.145 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:11 +0000 (0:00:00.089) 0:03:45.876 ********* 2021-05-12T08:58:11.213 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-05-12T08:58:11.214 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:11.234 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:11 +0000 (0:00:00.089) 0:03:45.965 ********* 2021-05-12T08:58:11.302 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-05-12T08:58:11.303 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:11.323 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:11 +0000 (0:00:00.089) 0:03:46.054 ********* 2021-05-12T08:58:11.391 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-05-12T08:58:11.393 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:11.413 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:11 +0000 (0:00:00.089) 0:03:46.144 ********* 2021-05-12T08:58:11.481 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-05-12T08:58:11.482 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:11.505 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:11 +0000 (0:00:00.091) 0:03:46.235 ********* 2021-05-12T08:58:11.562 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:11.585 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:11 +0000 (0:00:00.080) 0:03:46.316 ********* 2021-05-12T08:58:11.667 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:11.689 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:11 +0000 (0:00:00.103) 0:03:46.419 ********* 2021-05-12T08:58:11.757 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-05-12T08:58:11.759 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:11.779 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:11 +0000 (0:00:00.090) 0:03:46.510 ********* 2021-05-12T08:58:11.847 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:11.868 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:11 +0000 (0:00:00.089) 0:03:46.599 ********* 2021-05-12T08:58:11.936 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:11.958 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:11 +0000 (0:00:00.089) 0:03:46.688 ********* 2021-05-12T08:58:12.026 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-05-12T08:58:12.027 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:12.053 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:12 +0000 (0:00:00.095) 0:03:46.783 ********* 2021-05-12T08:58:12.123 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-05-12T08:58:12.125 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:12.148 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:12 +0000 (0:00:00.094) 0:03:46.878 ********* 2021-05-12T08:58:12.217 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-05-12T08:58:12.220 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:12.246 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:12 +0000 (0:00:00.098) 0:03:46.977 ********* 2021-05-12T08:58:12.328 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-05-12T08:58:12.330 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:12.350 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:12 +0000 (0:00:00.103) 0:03:47.081 ********* 2021-05-12T08:58:12.419 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T08:58:12.421 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-05-12T08:58:12.446 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:12 +0000 (0:00:00.095) 0:03:47.176 ********* 2021-05-12T08:58:12.526 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-05-12T08:58:12.528 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-05-12T08:58:12.537 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-05-12T08:58:12.538 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-05-12T08:58:12.549 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-05-12T08:58:12.551 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-05-12T08:58:12.560 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-05-12T08:58:12.562 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-05-12T08:58:12.586 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:12 +0000 (0:00:00.140) 0:03:47.317 ********* 2021-05-12T08:58:12.854 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-05-12T08:58:12.856 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003338", "end": "2021-05-12 08:58:12.832457", "rc": 0, "start": "2021-05-12 08:58:12.829119", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology-lock"]} 2021-05-12T08:58:12.882 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:12 +0000 (0:00:00.295) 0:03:47.612 ********* 2021-05-12T08:58:12.962 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-05-12T08:58:12.964 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:12.999 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:12 +0000 (0:00:00.116) 0:03:47.729 ********* 2021-05-12T08:58:13.079 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-05-12T08:58:13.081 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:13.101 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:13 +0000 (0:00:00.102) 0:03:47.832 ********* 2021-05-12T08:58:13.170 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-05-12T08:58:13.172 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:13.192 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:13 +0000 (0:00:00.091) 0:03:47.923 ********* 2021-05-12T08:58:13.272 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-05-12T08:58:13.274 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:13.300 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:13 +0000 (0:00:00.107) 0:03:48.031 ********* 2021-05-12T08:58:14.774 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-05-12T08:58:14.775 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T08:58:14.796 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:14 +0000 (0:00:01.495) 0:03:49.527 ********* 2021-05-12T08:58:14.875 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-05-12T08:58:14.876 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930278227968, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227118708, 'block_used': 12976400, 'inode_total': 61054976, 'inode_available': 60985794, 'inode_used': 69182, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227118708, "block_size": 4096, "block_total": 240095108, "block_used": 12976400, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985794, "inode_total": 61054976, "inode_used": 69182, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930278227968, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}} 2021-05-12T08:58:14.897 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:14 +0000 (0:00:00.101) 0:03:49.628 ********* 2021-05-12T08:58:15.132 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-05-12T08:58:15.134 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005174", "end": "2021-05-12 08:58:15.112950", "rc": 0, "start": "2021-05-12 08:58:15.107776", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T08:58:15.154 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:15 +0000 (0:00:00.256) 0:03:49.885 ********* 2021-05-12T08:58:15.234 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi028.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930278227968, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227118708, 'block_used': 12976400, 'inode_total': 61054976, 'inode_available': 60985794, 'inode_used': 69182, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227118708, "block_size": 4096, "block_total": 240095108, "block_used": 12976400, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985794, "inode_total": 61054976, "inode_used": 69182, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930278227968, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:15.258 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:15 +0000 (0:00:00.103) 0:03:49.988 ********* 2021-05-12T08:58:15.326 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-05-12T08:58:15.347 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:15 +0000 (0:00:00.089) 0:03:50.078 ********* 2021-05-12T08:58:17.132 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-05-12T08:58:17.134 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDc81jy5Ks1GtLOZc25z2qAYEpMjVz3AVAe'], 'uuids': ['c24934be-ebb9-4cc3-9d44-489db23c033c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.515072", "end": "2021-05-12 08:58:17.085177", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDc81jy5Ks1GtLOZc25z2qAYEpMjVz3AVAe"], "labels": [], "masters": [], "uuids": ["c24934be-ebb9-4cc3-9d44-489db23c033c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 08:58:15.570105", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T08:58:18.399 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-BmfEbz-3Twf-EaZP-omgQ-OcXo-V6nl-KKZex1', 'nvme-INTEL_SSDPED1D480GA_PHMB747401F4480DGN', 'nvme-nvme.8086-50484d42373437343031463434383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.058561", "end": "2021-05-12 08:58:18.365375", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-BmfEbz-3Twf-EaZP-omgQ-OcXo-V6nl-KKZex1", "nvme-INTEL_SSDPED1D480GA_PHMB747401F4480DGN", "nvme-nvme.8086-50484d42373437343031463434383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 08:58:17.306814", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-05-12T08:58:18.401 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T08:58:19.637 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDc85HrnEJHO5sjcghHxXAOaagld08KLVRQ'], 'uuids': ['838e29a7-7c5b-40e5-a1bb-419ad33430a9'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.049985", "end": "2021-05-12 08:58:19.610747", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDc85HrnEJHO5sjcghHxXAOaagld08KLVRQ"], "labels": [], "masters": [], "uuids": ["838e29a7-7c5b-40e5-a1bb-419ad33430a9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 08:58:18.560762", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T08:58:20.884 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDc7HoutNR3icspB3ksGZTASk23TlRmkJaz'], 'uuids': ['4f437ec7-b027-47ed-8fd1-9719520b3c12'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.065753", "end": "2021-05-12 08:58:20.860672", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDc7HoutNR3icspB3ksGZTASk23TlRmkJaz"], "labels": [], "masters": [], "uuids": ["4f437ec7-b027-47ed-8fd1-9719520b3c12"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 08:58:19.794919", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T08:58:22.136 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDcZH0C2Z5xC6VY2WjuASuhTRo3CZpBfoPS'], 'uuids': ['80bb3b4f-e88a-42f0-9fd9-a4d588758e53'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.054523", "end": "2021-05-12 08:58:22.102874", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDcZH0C2Z5xC6VY2WjuASuhTRo3CZpBfoPS"], "labels": [], "masters": [], "uuids": ["80bb3b4f-e88a-42f0-9fd9-a4d588758e53"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 08:58:21.048351", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T08:58:22.148 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J30L', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4J30L', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4J30L', 'scsi-35000c500871b80cc', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4J30L', 'wwn-0x5000c500871b80cc'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4J30L', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871b80cc', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J30L-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4J30L-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4J30L-part1', 'scsi-35000c500871b80cc-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4J30L-part1', 'wwn-0x5000c500871b80cc-part1'], 'uuids': ['a9bc468f-7d76-441a-b17b-cd059807719c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J30L", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4J30L", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4J30L", "scsi-35000c500871b80cc", "scsi-SATA_ST1000NM0033-9ZM_Z1W4J30L", "wwn-0x5000c500871b80cc"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J30L-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4J30L-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-05-12T08:58:22.150 INFO:teuthology.task.ansible.out:Z1W4J30L-part1", "scsi-35000c500871b80cc-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4J30L-part1", "wwn-0x5000c500871b80cc-part1"], "labels": [], "masters": [], "uuids": ["a9bc468f-7d76-441a-b17b-cd059807719c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J30L", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b80cc"}}, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:23.398 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDcocPYskfnEJS2RqDDj9xjKpcrSiTbBCFP'], 'uuids': ['ba064778-ddcf-447e-a690-91d67526e70e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.061783", "end": "2021-05-12 08:58:23.368608", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDcocPYskfnEJS2RqDDj9xjKpcrSiTbBCFP"], "labels": [], "masters": [], "uuids": ["ba064778-ddcf-447e-a690-91d67526e70e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 08:58:22.306825", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-05-12T08:58:23.426 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:23 +0000 (0:00:08.078) 0:03:58.156 ********* 2021-05-12T08:58:23.785 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDc81jy5Ks1GtLOZc25z2qAYEpMjVz3AVAe'], 'uuids': ['c24934be-ebb9-4cc3-9d44-489db23c033c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.097042", "end": "2021-05-12 08:58:23.761293", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDc81jy5Ks1GtLOZc25z2qAYEpMjVz3AVAe"], "labels": [], "masters": [], "uuids": ["c24934be-ebb9-4cc3-9d44-489db23c033c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 08:58:23.664251", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0943324 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0943324 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T08:58:24.004 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-BmfEbz-3Twf-EaZP-omgQ-OcXo-V6nl-KKZex1', 'nvme-INTEL_SSDPED1D480GA_PHMB747401F4480DGN', 'nvme-nvme.8086-50484d42373437343031463434383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.036637", "end": "2021-05-12 08:58:23.977795", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-BmfEbz-3Twf-EaZP-omgQ-OcXo-V6nl-KKZex1", "nvme-INTEL_SSDPED1D480GA_PHMB747401F4480DGN", "nvme-nvme.8086-50484d42373437343031463434383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 08:58:23.941158", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0339111 s, 3.4 GB/s", "stderr_lines": [" 2021-05-12T08:58:24.005 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0339111 s, 3.4 GB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T08:58:24.274 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDc85HrnEJHO5sjcghHxXAOaagld08KLVRQ'], 'uuids': ['838e29a7-7c5b-40e5-a1bb-419ad33430a9'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.094516", "end": "2021-05-12 08:58:24.248606", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDc85HrnEJHO5sjcghHxXAOaagld08KLVRQ"], "labels": [], "masters": [], "uuids": ["838e29a7-7c5b-40e5-a1bb-419ad33430a9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 08:58:24.154090", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0918079 s, 1.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0918079 s, 1.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T08:58:24.544 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDc7HoutNR3icspB3ksGZTASk23TlRmkJaz'], 'uuids': ['4f437ec7-b027-47ed-8fd1-9719520b3c12'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.095393", "end": "2021-05-12 08:58:24.517984", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDc7HoutNR3icspB3ksGZTASk23TlRmkJaz"], "labels": [], "masters": [], "uuids": ["4f437ec7-b027-47ed-8fd1-9719520b3c12"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 08:58:24.422591", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0926626 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0926626 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T08:58:24.814 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDcZH0C2Z5xC6VY2WjuASuhTRo3CZpBfoPS'], 'uuids': ['80bb3b4f-e88a-42f0-9fd9-a4d588758e53'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.095471", "end": "2021-05-12 08:58:24.788437", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDcZH0C2Z5xC6VY2WjuASuhTRo3CZpBfoPS"], "labels": [], "masters": [], "uuids": ["80bb3b4f-e88a-42f0-9fd9-a4d588758e53"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 08:58:24.692966", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0927594 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0927594 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T08:58:24.816 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J30L', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4J30L', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4J30L', 'scsi-35000c500871b80cc', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4J30L', 'wwn-0x5000c500871b80cc'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4J30L', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871b80cc', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J30L-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4J30L-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4J30L-part1', 'scsi-35000c500871b80cc-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4J30L-part1', 'wwn-0x5000c500871b80cc-part1'], 'uuids': ['a9bc468f-7d76-441a-b17b-cd059807719c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J30L", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4J30L", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4J30L", "scsi-35000c500871b80cc", "scsi-SATA_ST1000NM0033-9ZM_Z1W4J30L", "wwn-0x5000c500871b80cc"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J30L-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4J30L-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-05-12T08:58:24.817 INFO:teuthology.task.ansible.out:Z1W4J30L-part1", "scsi-35000c500871b80cc-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4J30L-part1", "wwn-0x5000c500871b80cc-part1"], "labels": [], "masters": [], "uuids": ["a9bc468f-7d76-441a-b17b-cd059807719c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J30L", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b80cc"}}, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:25.086 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDcocPYskfnEJS2RqDDj9xjKpcrSiTbBCFP'], 'uuids': ['ba064778-ddcf-447e-a690-91d67526e70e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.096469", "end": "2021-05-12 08:58:25.065462", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-DpvJNMGlUwcIyk9IR0clvnJJdeAm2nDcocPYskfnEJS2RqDDj9xjKpcrSiTbBCFP"], "labels": [], "masters": [], "uuids": ["ba064778-ddcf-447e-a690-91d67526e70e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-05-12 08:58:24.968993", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0937832 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0937832 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-05-12T08:58:25.125 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:25 +0000 (0:00:01.699) 0:03:59.856 ********* 2021-05-12T08:58:25.421 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi028.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.063033", "end": "2021-05-12 08:58:25.355312", "rc": 0, "start": "2021-05-12 08:58:25.292279", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T08:58:25.443 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:25 +0000 (0:00:00.317) 0:04:00.173 ********* 2021-05-12T08:58:25.682 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-05-12T08:58:25.684 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.049506", "end": "2021-05-12 08:58:25.656982", "rc": 0, "start": "2021-05-12 08:58:25.607476", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T08:58:25.704 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:25 +0000 (0:00:00.260) 0:04:00.434 ********* 2021-05-12T08:58:25.762 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-05-12T08:58:25.820 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:25 +0000 (0:00:00.115) 0:04:00.550 ********* 2021-05-12T08:58:25.846 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:25.870 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:25 +0000 (0:00:00.050) 0:04:00.601 ********* 2021-05-12T08:58:25.896 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-05-12T08:58:25.898 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:25.918 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:25 +0000 (0:00:00.047) 0:04:00.648 ********* 2021-05-12T08:58:25.944 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-05-12T08:58:25.946 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:25.968 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:25 +0000 (0:00:00.049) 0:04:00.698 ********* 2021-05-12T08:58:25.994 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-05-12T08:58:25.996 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:26.016 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:26 +0000 (0:00:00.048) 0:04:00.747 ********* 2021-05-12T08:58:26.206 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-05-12T08:58:26.208 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-05-12T08:58:26.231 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:26 +0000 (0:00:00.214) 0:04:00.962 ********* 2021-05-12T08:58:26.269 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi028.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930278227968, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227118708, 'block_used': 12976400, 'inode_total': 61054976, 'inode_available': 60985794, 'inode_used': 69182, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227118708, "block_size": 4096, "block_total": 240095108, "block_used": 12976400, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985794, "inode_total": 61054976, "inode_used": 69182, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930278227968, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:26.292 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:26 +0000 (0:00:00.059) 0:04:01.021 ********* 2021-05-12T08:58:26.318 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-05-12T08:58:26.320 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:26.340 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:26 +0000 (0:00:00.048) 0:04:01.070 ********* 2021-05-12T08:58:26.366 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T08:58:26.368 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:26.388 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:26 +0000 (0:00:00.048) 0:04:01.119 ********* 2021-05-12T08:58:26.420 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:26.437 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:26 +0000 (0:00:00.048) 0:04:01.167 ********* 2021-05-12T08:58:26.464 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:26.486 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:26 +0000 (0:00:00.049) 0:04:01.216 ********* 2021-05-12T08:58:26.514 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-05-12T08:58:26.516 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:26.536 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:26 +0000 (0:00:00.049) 0:04:01.266 ********* 2021-05-12T08:58:26.562 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-05-12T08:58:26.564 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:26.584 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:26 +0000 (0:00:00.048) 0:04:01.315 ********* 2021-05-12T08:58:27.288 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-05-12T08:58:27.289 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-05-12T08:58:27.309 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:27 +0000 (0:00:00.725) 0:04:02.040 ********* 2021-05-12T08:58:27.938 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-05-12T08:58:27.940 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T08:58:28.263 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T08:58:28.580 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T08:58:28.897 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T08:58:29.242 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-05-12T08:58:29.266 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:29 +0000 (0:00:01.956) 0:04:03.996 ********* 2021-05-12T08:58:29.461 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-05-12T08:58:29.483 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:29 +0000 (0:00:00.216) 0:04:04.213 ********* 2021-05-12T08:58:29.698 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-05-12T08:58:29.700 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-12T08:58:29.877 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-12T08:58:30.055 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-12T08:58:30.228 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-05-12T08:58:30.238 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:30.263 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:30 +0000 (0:00:00.780) 0:04:04.993 ********* 2021-05-12T08:58:30.290 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-05-12T08:58:30.292 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:30.316 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:30 +0000 (0:00:00.053) 0:04:05.046 ********* 2021-05-12T08:58:30.343 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-05-12T08:58:30.345 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:30.374 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:30 +0000 (0:00:00.057) 0:04:05.104 ********* 2021-05-12T08:58:30.580 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-05-12T08:58:30.583 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-12T08:58:30.603 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:30 +0000 (0:00:00.228) 0:04:05.333 ********* 2021-05-12T08:58:30.639 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-05-12T08:58:30.641 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-05-12T08:58:30.663 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:30 +0000 (0:00:00.060) 0:04:05.393 ********* 2021-05-12T08:58:31.157 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-05-12T08:58:31.159 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true} 2021-05-12T08:58:31.182 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:31 +0000 (0:00:00.519) 0:04:05.913 ********* 2021-05-12T08:58:31.750 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-05-12T08:58:31.752 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-05-12T08:58:31.772 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:31 +0000 (0:00:00.589) 0:04:06.502 ********* 2021-05-12T08:58:31.798 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:31.823 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:31 +0000 (0:00:00.051) 0:04:06.554 ********* 2021-05-12T08:58:31.850 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-05-12T08:58:31.852 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:31.872 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:31 +0000 (0:00:00.047) 0:04:06.602 ********* 2021-05-12T08:58:31.898 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-05-12T08:58:31.899 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:31.920 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:31 +0000 (0:00:00.048) 0:04:06.650 ********* 2021-05-12T08:58:31.957 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-05-12T08:58:31.959 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:31.988 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:31 +0000 (0:00:00.067) 0:04:06.718 ********* 2021-05-12T08:58:32.018 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-05-12T08:58:32.019 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:32.046 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:32 +0000 (0:00:00.058) 0:04:06.777 ********* 2021-05-12T08:58:32.073 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-05-12T08:58:32.075 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:32.095 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:32 +0000 (0:00:00.048) 0:04:06.825 ********* 2021-05-12T08:58:32.121 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-05-12T08:58:32.123 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:32.143 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:32 +0000 (0:00:00.047) 0:04:06.873 ********* 2021-05-12T08:58:32.169 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:32.190 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:32 +0000 (0:00:00.047) 0:04:06.921 ********* 2021-05-12T08:58:32.224 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-05-12T08:58:32.226 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:32.246 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:32 +0000 (0:00:00.055) 0:04:06.976 ********* 2021-05-12T08:58:32.272 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-05-12T08:58:32.274 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:32.296 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:32 +0000 (0:00:00.050) 0:04:07.026 ********* 2021-05-12T08:58:32.322 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-05-12T08:58:32.324 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:32.344 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:32 +0000 (0:00:00.047) 0:04:07.074 ********* 2021-05-12T08:58:32.370 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-05-12T08:58:32.372 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:32.391 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:32 +0000 (0:00:00.047) 0:04:07.122 ********* 2021-05-12T08:58:32.418 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-05-12T08:58:32.420 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:32.439 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:32 +0000 (0:00:00.047) 0:04:07.169 ********* 2021-05-12T08:58:32.467 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:32.487 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:32 +0000 (0:00:00.048) 0:04:07.218 ********* 2021-05-12T08:58:32.514 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:32.535 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:32 +0000 (0:00:00.047) 0:04:07.266 ********* 2021-05-12T08:58:33.006 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-05-12T08:58:33.007 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-05-12T08:58:33.027 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:33 +0000 (0:00:00.492) 0:04:07.758 ********* 2021-05-12T08:58:33.334 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-05-12T08:58:33.336 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-05-12 08:54:10 UTC", "ActiveEnterTimestampMonotonic": "15951373", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target ntpd.service sntp.service basic.target systemd-tmpfiles-setup.service system.slice systemd-journald.socket -.mount ntpdate.service tmp.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-05-12 08:54:09 UTC", "AssertTimestampMonotonic": "14765670", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-05-12 08:54:09 UTC", "ConditionTimestampMonotonic": "14765632", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target ntpd.service systemd-timesyncd.service", "ControlGro 2021-05-12T08:58:33.338 INFO:teuthology.task.ansible.out:up": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "884", "ExecMainStartTimestamp": "Wed 2021-05-12 08:54:10 UTC", "ExecMainStartTimestampMonotonic": "15865816", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-05-12 08:54:09 UTC", "InactiveExitTimestampMonotonic": "14814710", "InvocationID": "d3a1324f7cca45909889fddb87881ca1", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity" 2021-05-12T08:58:33.340 INFO:teuthology.task.ansible.out:, "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127277", "LimitNPROCSoft": "127277", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127277", "LimitSIGPENDINGSoft": "127277", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "884", "MemoryAccounting": "yes", "MemoryCurrent": "3530752", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp", "Restart": "no 2021-05-12T08:58:33.341 INFO:teuthology.task.ansible.out:", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-05-12 08:54:10 UTC", "StateChangeTimestampMonotonic": "15951373", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203643", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-05-12 08:54:10 UTC", "WatchdogTimestampMonotonic": "15865838", "WatchdogUSec": "0"}} 2021-05-12T08:58:33.359 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:33 +0000 (0:00:00.331) 0:04:08.090 ********* 2021-05-12T08:58:33.387 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T08:58:33.389 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-05-12T08:58:33.409 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:33 +0000 (0:00:00.049) 0:04:08.139 ********* 2021-05-12T08:58:33.446 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T08:58:33.447 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-05-12T08:58:33.468 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:33 +0000 (0:00:00.058) 0:04:08.198 ********* 2021-05-12T08:58:33.494 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-05-12T08:58:33.497 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:33.519 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:33 +0000 (0:00:00.051) 0:04:08.249 ********* 2021-05-12T08:58:33.548 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-05-12T08:58:33.550 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:33.574 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:33 +0000 (0:00:00.054) 0:04:08.304 ********* 2021-05-12T08:58:33.610 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-05-12T08:58:33.612 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:33.632 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:33 +0000 (0:00:00.058) 0:04:08.363 ********* 2021-05-12T08:58:33.834 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-05-12T08:58:33.836 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-05-12T08:58:33.856 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:33 +0000 (0:00:00.224) 0:04:08.587 ********* 2021-05-12T08:58:34.337 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-05-12T08:58:34.339 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-05-12T08:58:34.359 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:34 +0000 (0:00:00.502) 0:04:09.089 ********* 2021-05-12T08:58:35.756 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-05-12T08:58:35.757 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-05-12T08:58:35.779 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:35 +0000 (0:00:01.419) 0:04:10.509 ********* 2021-05-12T08:58:35.816 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-05-12T08:58:35.818 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:35.838 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:35 +0000 (0:00:00.059) 0:04:10.569 ********* 2021-05-12T08:58:35.865 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-05-12T08:58:35.867 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-05-12T08:58:35.887 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:35 +0000 (0:00:00.048) 0:04:10.617 ********* 2021-05-12T08:58:36.069 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-05-12T08:58:36.071 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-05-12T08:58:36.091 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:36 +0000 (0:00:00.203) 0:04:10.821 ********* 2021-05-12T08:58:36.117 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-05-12T08:58:36.119 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:36.141 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:36 +0000 (0:00:00.049) 0:04:10.871 ********* 2021-05-12T08:58:36.177 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-05-12T08:58:36.179 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-05-12T08:58:36.199 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:36 +0000 (0:00:00.058) 0:04:10.929 ********* 2021-05-12T08:58:36.225 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-05-12T08:58:36.227 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:36.250 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:36 +0000 (0:00:00.050) 0:04:10.980 ********* 2021-05-12T08:58:36.455 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-05-12T08:58:36.457 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-05-12T08:58:36.480 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:36 +0000 (0:00:00.229) 0:04:11.210 ********* 2021-05-12T08:58:36.683 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi028.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-05-12T08:58:36.704 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:36 +0000 (0:00:00.224) 0:04:11.435 ********* 2021-05-12T08:58:36.731 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-05-12T08:58:36.733 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:36.752 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:36 +0000 (0:00:00.048) 0:04:11.483 ********* 2021-05-12T08:58:36.800 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-05-12T08:58:36.802 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi028.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-05-12T08:58:36.812 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi028.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-05-12T08:58:36.823 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-05-12T08:58:36.845 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:36 +0000 (0:00:00.092) 0:04:11.576 ********* 2021-05-12T08:58:36.893 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-05-12T08:58:36.895 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi028.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:36.905 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:36.907 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi028.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:36.929 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:36 +0000 (0:00:00.083) 0:04:11.659 ********* 2021-05-12T08:58:36.955 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-05-12T08:58:36.957 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:36.979 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:36 +0000 (0:00:00.050) 0:04:11.709 ********* 2021-05-12T08:58:37.006 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-05-12T08:58:37.008 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:37.033 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:37 +0000 (0:00:00.054) 0:04:11.764 ********* 2021-05-12T08:58:37.059 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:37.083 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:37 +0000 (0:00:00.049) 0:04:11.814 ********* 2021-05-12T08:58:37.110 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-05-12T08:58:37.111 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:37.131 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:37 +0000 (0:00:00.048) 0:04:11.862 ********* 2021-05-12T08:58:37.158 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-05-12T08:58:37.159 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:37.181 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:37 +0000 (0:00:00.049) 0:04:11.911 ********* 2021-05-12T08:58:37.207 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-05-12T08:58:37.209 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:37.230 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:37 +0000 (0:00:00.049) 0:04:11.961 ********* 2021-05-12T08:58:37.278 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-05-12T08:58:37.280 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:37.305 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:37 +0000 (0:00:00.073) 0:04:12.034 ********* 2021-05-12T08:58:37.330 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-05-12T08:58:37.332 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:37.353 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:37 +0000 (0:00:00.048) 0:04:12.083 ********* 2021-05-12T08:58:37.567 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-05-12T08:58:37.569 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005379", "end": "2021-05-12 08:58:37.536557", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-05-12 08:58:37.531178", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T08:58:38.742 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:01.004411", "end": "2021-05-12 08:58:38.714126", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-05-12 08:58:37.709715", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T08:58:38.928 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003022", "end": "2021-05-12 08:58:38.903980", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-05-12 08:58:38.900958", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-05-12T08:58:39.105 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003314", "end": "2021-05-12 08:58:39.078877", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-05-12 08:58:39.075563", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T08:58:39.127 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:39 +0000 (0:00:01.774) 0:04:13.857 ********* 2021-05-12T08:58:39.153 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-05-12T08:58:39.175 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:39 +0000 (0:00:00.047) 0:04:13.905 ********* 2021-05-12T08:58:39.378 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-05-12T08:58:39.445 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-05-12T08:58:39.462 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:39 +0000 (0:00:00.286) 0:04:14.192 ********* 2021-05-12T08:58:40.246 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi028.front.sepia.ceph.com] 2021-05-12T08:58:40.292 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:40 +0000 (0:00:00.830) 0:04:15.022 ********* 2021-05-12T08:58:40.318 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-12T08:58:40.320 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:40.340 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:40 +0000 (0:00:00.047) 0:04:15.070 ********* 2021-05-12T08:58:40.366 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi028.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-05-12T08:58:40.388 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:40 +0000 (0:00:00.047) 0:04:15.118 ********* 2021-05-12T08:58:40.424 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-05-12T08:58:40.426 INFO:teuthology.task.ansible.out:ok: [smithi028.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-05-12T08:58:40.446 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:40 +0000 (0:00:00.058) 0:04:15.177 ********* 2021-05-12T08:58:41.850 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** ok: [smithi028.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-05-12T08:58:41.873 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:41 +0000 (0:00:01.426) 0:04:16.603 ********* 2021-05-12T08:58:41.899 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi028.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-05-12T08:58:41.921 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:41 +0000 (0:00:00.047) 0:04:16.651 ********* 2021-05-12T08:58:41.947 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-05-12T08:58:41.949 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:41.969 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:41 +0000 (0:00:00.048) 0:04:16.700 ********* 2021-05-12T08:58:42.193 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-05-12T08:58:42.194 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-12T08:58:42.370 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-05-12T08:58:42.396 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:42 +0000 (0:00:00.426) 0:04:17.126 ********* 2021-05-12T08:58:42.943 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-05-12T08:58:42.947 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1620809922.4594524-24546-198644174918499/source", "state": "file", "uid": 0} 2021-05-12T08:58:43.471 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1620809922.9788675-24546-238478954692645/source", "state": "file", "uid": 0} 2021-05-12T08:58:43.494 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:43 +0000 (0:00:01.098) 0:04:18.225 ********* 2021-05-12T08:58:46.672 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-cjtnqh36-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-cjtnqh36-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl" 2021-05-12T08:58:46.674 INFO:teuthology.task.ansible.out:, "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-05-12T08:58:46.694 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:46 +0000 (0:00:03.200) 0:04:21.425 ********* 2021-05-12T08:58:46.722 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:46.746 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:46 +0000 (0:00:00.051) 0:04:21.476 ********* 2021-05-12T08:58:46.773 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:46.795 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:46 +0000 (0:00:00.049) 0:04:21.526 ********* 2021-05-12T08:58:47.155 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-05-12T08:58:47.156 INFO:teuthology.task.ansible.out:changed: [smithi028.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.166218", "end": "2021-05-12 08:58:47.135040", "rc": 0, "start": "2021-05-12 08:58:46.968822", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-05-12T08:58:47.177 INFO:teuthology.task.ansible.out:Wednesday 12 May 2021 08:58:47 +0000 (0:00:00.381) 0:04:21.907 ********* 2021-05-12T08:58:47.203 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-05-12T08:58:47.204 INFO:teuthology.task.ansible.out:skipping: [smithi028.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-05-12T08:58:47.269 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-05-12T08:58:47.270 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-05-12T08:58:47.272 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi028.front.sepia.ceph.com : ok=127 changed=42 unreachable=0 failed=0 skipped=185 rescued=0 ignored=3 Wednesday 12 May 2021 08:58:47 +0000 (0:00:00.094) 0:04:22.002 ********* =============================================================================== 2021-05-12T08:58:47.273 INFO:teuthology.task.ansible.out:common : nrpe - Ensure SELinux policy is up to date -------------------- 39.70s Create all admin users with sudo access. ------------------------------- 30.06s users : Update authorized_keys using the keys repo --------------------- 29.59s common : nrpe - Load SELinux policy package ---------------------------- 17.26s common : nrpe - Remove SELinux policy package -------------------------- 17.23s Remove revoked users --------------------------------------------------- 11.48s testnode : Zap all non-root disks --------------------------------------- 8.08s testnode : Install packages --------------------------------------------- 6.98s common : Install nrpe package and dependencies (RHEL/CentOS) ------------ 5.48s common : Install Kerberos Packages (RedHat) ----------------------------- 4.16s container-host : Install registries-conf-ctl ---------------------------- 3.20s testnode : Ensure ceph packages are not present. ------------------------ 2.99s Gathering Facts --------------------------------------------------------- 2.90s testnode : Ensure ceph dependency packages are not present. ------------- 2.70s testnode : Rebuild rpmdb ------------------------------------------------ 2.55s testnode : Create logical volume(s) ------------------------------------- 1.96s testnode : List any leftover Ceph artifacts from previous jobs ---------- 1.77s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.70s Gathering Facts --------------------------------------------------------- 1.69s users : Update authorized_keys for each user with literal keys ---------- 1.63s 2021-05-12T08:58:47.585 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com')] 2021-05-12T08:58:47.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-12T08:58:47.588 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-05-12T08:58:47.662 DEBUG:teuthology.orchestra.run.smithi028:> true 2021-05-12T08:58:47.755 INFO:teuthology.run_tasks:Running task clock... 2021-05-12T08:58:47.770 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-05-12T08:58:47.770 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-05-12T08:58:47.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-05-12T08:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-05-12T08:58:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-05-12T08:58:47.878 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2021-05-12T08:58:47.889 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2021-05-12T08:58:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-05-12T08:58:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-05-12T08:58:47.980 INFO:teuthology.orchestra.run.smithi028.stderr:bash: ntpq: command not found 2021-05-12T08:58:47.983 INFO:teuthology.orchestra.run.smithi028.stdout:210 Number of sources = 4 2021-05-12T08:58:47.983 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-05-12T08:58:47.983 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2021-05-12T08:58:47.984 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-12T08:58:47.984 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-12T08:58:47.984 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-12T08:58:47.984 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-05-12T08:58:47.985 INFO:teuthology.run_tasks:Running task install... 2021-05-12T08:58:47.997 DEBUG:teuthology.task.install:project ceph 2021-05-12T08:58:47.998 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'a8c673d555488beec9ccda5812ba822e69c12ea9'}} 2021-05-12T08:58:47.998 DEBUG:teuthology.task.install:config {'sha1': 'a8c673d555488beec9ccda5812ba822e69c12ea9'} 2021-05-12T08:58:47.998 INFO:teuthology.task.install:Using flavor: basic 2021-05-12T08:58:48.006 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-05-12T08:58:48.007 INFO:teuthology.task.install:extra packages: [] 2021-05-12T08:58:48.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo lsb_release -is 2021-05-12T08:58:48.057 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS 2021-05-12T08:58:48.059 DEBUG:teuthology.misc:System to be installed: CentOS 2021-05-12T08:58:48.059 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a8c673d555488beec9ccda5812ba822e69c12ea9', 'tag': None, 'wait_for_package': False} 2021-05-12T08:58:48.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo lsb_release -is 2021-05-12T08:58:48.132 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS 2021-05-12T08:58:48.134 DEBUG:teuthology.misc:System to be installed: CentOS 2021-05-12T08:58:48.134 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a8c673d555488beec9ccda5812ba822e69c12ea9 2021-05-12T08:58:48.299 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-05-12-1357/a8c673d555488beec9ccda5812ba822e69c12ea9/centos/8/flavors/default/ 2021-05-12T08:58:48.299 INFO:teuthology.task.install.rpm:Package version is 17.0.0-4095.ga8c673d5 2021-05-12T08:58:48.418 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-05-12-1357/a8c673d555488beec9ccda5812ba822e69c12ea9/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-05-12-1357/a8c673d555488beec9ccda5812ba822e69c12ea9/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-05-12-1357/a8c673d555488beec9ccda5812ba822e69c12ea9/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-05-12T08:58:48.419 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2021-05-12T08:58:48.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-05-12T08:58:48.447 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-05-12T08:58:48.447 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a8c673d555488beec9ccda5812ba822e69c12ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-05-12T08:58:48.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-05-12T08:58:48.591 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-05-12T08:58:48.656 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2021-05-12T08:58:48.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2021-05-12T08:58:49.049 INFO:teuthology.orchestra.run.smithi028.stdout:72 files removed 2021-05-12T08:58:49.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw 2021-05-12T08:58:49.765 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 312 kB/s | 77 kB 00:00 2021-05-12T08:58:50.003 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 76 kB/s | 16 kB 00:00 2021-05-12T08:58:50.229 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 7.1 kB/s | 1.5 kB 00:00 2021-05-12T08:58:50.396 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-8 - AppStream 42 MB/s | 6.3 MB 00:00 2021-05-12T08:58:51.538 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-8 - Base 25 MB/s | 2.3 MB 00:00 2021-05-12T08:58:52.002 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-8 - Extras 146 kB/s | 9.6 kB 00:00 2021-05-12T08:58:52.650 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Linux 8 - AppStream 11 MB/s | 6.3 MB 00:00 2021-05-12T08:58:53.981 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Linux 8 - BaseOS 9.4 MB/s | 2.3 MB 00:00 2021-05-12T08:58:54.684 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Linux 8 - Extras 31 kB/s | 9.6 kB 00:00 2021-05-12T08:58:55.435 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 13 MB/s | 9.4 MB 00:00 2021-05-12T08:58:57.515 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 3.6 MB/s | 24 kB 00:00 2021-05-12T08:58:59.248 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T08:58:59.267 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T08:58:59.268 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2021-05-12T08:58:59.268 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T08:58:59.269 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T08:58:59.269 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 9.9 M 2021-05-12T08:58:59.269 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2021-05-12T08:58:59.269 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 3.3 M 2021-05-12T08:58:59.270 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 3.8 M 2021-05-12T08:58:59.270 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2021-05-12T08:58:59.270 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 5.4 M 2021-05-12T08:58:59.270 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 19 M 2021-05-12T08:58:59.270 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 23 k 2021-05-12T08:58:59.271 INFO:teuthology.orchestra.run.smithi028.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-05-12T08:58:59.271 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-05-12T08:58:59.271 INFO:teuthology.orchestra.run.smithi028.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-05-12T08:58:59.271 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 757 k 2021-05-12T08:58:59.272 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-05-12T08:58:59.272 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.9.0-2.el8 CentOS-Base 47 k 2021-05-12T08:58:59.272 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 457 k 2021-05-12T08:58:59.272 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-05-12T08:58:59.273 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 3.8 M 2021-05-12T08:58:59.273 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-05-12T08:58:59.273 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-05-12T08:58:59.273 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse 2021-05-12T08:58:59.274 INFO:teuthology.orchestra.run.smithi028.stdout: x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 42 k 2021-05-12T08:58:59.274 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 77 k 2021-05-12T08:58:59.274 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 177 k 2021-05-12T08:58:59.274 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-05-12T08:58:59.274 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 364 k 2021-05-12T08:58:59.275 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 345 k 2021-05-12T08:58:59.275 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 109 k 2021-05-12T08:58:59.275 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:58:59.275 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T08:58:59.276 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T08:58:59.276 INFO:teuthology.orchestra.run.smithi028.stdout:Install 22 Packages 2021-05-12T08:58:59.276 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 2 Packages 2021-05-12T08:58:59.276 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:58:59.277 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 49 M 2021-05-12T08:58:59.277 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T08:59:00.477 INFO:teuthology.orchestra.run.smithi028.stdout:(1/24): ceph-base-17.0.0-4095.ga8c673d5.el8.x86 4.5 MB/s | 5.4 MB 00:01 2021-05-12T08:59:00.507 INFO:teuthology.orchestra.run.smithi028.stdout:(2/24): ceph-selinux-17.0.0-4095.ga8c673d5.el8. 789 kB/s | 23 kB 00:00 2021-05-12T08:59:00.620 INFO:teuthology.orchestra.run.smithi028.stdout:(3/24): libcephfs2-17.0.0-4095.ga8c673d5.el8.x8 6.5 MB/s | 757 kB 00:00 2021-05-12T08:59:00.697 INFO:teuthology.orchestra.run.smithi028.stdout:(4/24): libradosstriper1-17.0.0-4095.ga8c673d5. 5.8 MB/s | 457 kB 00:00 2021-05-12T08:59:00.895 INFO:teuthology.orchestra.run.smithi028.stdout:(5/24): ceph-radosgw-17.0.0-4095.ga8c673d5.el8. 6.1 MB/s | 9.9 MB 00:01 2021-05-12T08:59:00.941 INFO:teuthology.orchestra.run.smithi028.stdout:(6/24): python3-ceph-argparse-17.0.0-4095.ga8c6 933 kB/s | 42 kB 00:00 2021-05-12T08:59:00.982 INFO:teuthology.orchestra.run.smithi028.stdout:(7/24): python3-ceph-common-17.0.0-4095.ga8c673 1.8 MB/s | 77 kB 00:00 2021-05-12T08:59:01.029 INFO:teuthology.orchestra.run.smithi028.stdout:(8/24): python3-cephfs-17.0.0-4095.ga8c673d5.el 3.7 MB/s | 177 kB 00:00 2021-05-12T08:59:01.087 INFO:teuthology.orchestra.run.smithi028.stdout:(9/24): python3-rados-17.0.0-4095.ga8c673d5.el8 6.2 MB/s | 364 kB 00:00 2021-05-12T08:59:01.125 INFO:teuthology.orchestra.run.smithi028.stdout:(10/24): librgw2-17.0.0-4095.ga8c673d5.el8.x86_ 9.0 MB/s | 3.8 MB 00:00 2021-05-12T08:59:01.136 INFO:teuthology.orchestra.run.smithi028.stdout:(11/24): python3-rbd-17.0.0-4095.ga8c673d5.el8. 6.8 MB/s | 345 kB 00:00 2021-05-12T08:59:01.166 INFO:teuthology.orchestra.run.smithi028.stdout:(12/24): python3-rgw-17.0.0-4095.ga8c673d5.el8. 2.6 MB/s | 109 kB 00:00 2021-05-12T08:59:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:(13/24): librdkafka-0.11.4-1.el8.x86_64.rpm 3.1 MB/s | 353 kB 00:00 2021-05-12T08:59:01.251 INFO:teuthology.orchestra.run.smithi028.stdout:(14/24): python3-prettytable-0.7.2-14.el8.noarc 33 MB/s | 44 kB 00:00 2021-05-12T08:59:01.253 INFO:teuthology.orchestra.run.smithi028.stdout:(15/24): librabbitmq-0.9.0-2.el8.x86_64.rpm 23 MB/s | 47 kB 00:00 2021-05-12T08:59:01.400 INFO:teuthology.orchestra.run.smithi028.stdout:(16/24): ceph-common-17.0.0-4095.ga8c673d5.el8. 9.0 MB/s | 19 MB 00:02 2021-05-12T08:59:01.401 INFO:teuthology.orchestra.run.smithi028.stdout:(17/24): fmt-6.2.1-1.el8.x86_64.rpm 783 kB/s | 116 kB 00:00 2021-05-12T08:59:01.455 INFO:teuthology.orchestra.run.smithi028.stdout:(18/24): leveldb-1.22-1.el8.x86_64.rpm 3.3 MB/s | 181 kB 00:00 2021-05-12T08:59:01.483 INFO:teuthology.orchestra.run.smithi028.stdout:(19/24): liboath-2.6.2-3.el8.x86_64.rpm 2.1 MB/s | 59 kB 00:00 2021-05-12T08:59:01.512 INFO:teuthology.orchestra.run.smithi028.stdout:(20/24): libunwind-1.3.1-3.el8.x86_64.rpm 2.7 MB/s | 75 kB 00:00 2021-05-12T08:59:01.556 INFO:teuthology.orchestra.run.smithi028.stdout:(21/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 666 kB/s | 259 kB 00:00 2021-05-12T08:59:01.562 INFO:teuthology.orchestra.run.smithi028.stdout:(22/24): gperftools-libs-2.7-9.el8.x86_64.rpm 1.9 MB/s | 306 kB 00:00 2021-05-12T08:59:01.796 INFO:teuthology.orchestra.run.smithi028.stdout:(23/24): librbd1-17.0.0-4095.ga8c673d5.el8.x86_ 16 MB/s | 3.8 MB 00:00 2021-05-12T08:59:01.806 INFO:teuthology.orchestra.run.smithi028.stdout:(24/24): librados2-17.0.0-4095.ga8c673d5.el8.x8 11 MB/s | 3.3 MB 00:00 2021-05-12T08:59:01.811 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T08:59:01.811 INFO:teuthology.orchestra.run.smithi028.stdout:Total 19 MB/s | 49 MB 00:02 2021-05-12T08:59:01.815 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T08:59:01.865 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T08:59:01.865 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T08:59:02.264 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T08:59:02.267 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T08:59:03.309 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T08:59:03.448 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2021-05-12T08:59:03.784 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2021-05-12T08:59:03.946 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-05-12T08:59:04.365 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-05-12T08:59:04.593 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:17.0.0-4095.ga8c673d5.el8.x86_64 4/26 2021-05-12T08:59:04.691 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:17.0.0-4095.ga8c673d5.el8.x86_64 4/26 2021-05-12T08:59:04.922 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:17.0.0-4095.ga8c673d5.el8.x86_64 5/26 2021-05-12T08:59:05.102 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:17.0.0-4095.ga8c673d5.el8.x86_64 6/26 2021-05-12T08:59:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:17.0.0-4095.ga8c673d5.el8.x86_64 6/26 2021-05-12T08:59:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:17.0.0-4095.ga8c673d5.el8.x86_64 7/26 2021-05-12T08:59:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:17.0.0-4095.ga8c673d5.el8.x86_64 7/26 2021-05-12T08:59:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 8/26 2021-05-12T08:59:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2021-05-12T08:59:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2021-05-12T08:59:06.818 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:17.0.0-4095.ga8c673d5.el8.x86_64 10/26 2021-05-12T08:59:06.902 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:17.0.0-4095.ga8c673d5.el8.x86_64 10/26 2021-05-12T08:59:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2021-05-12T08:59:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:17.0.0-4095.ga8c673d5.el8. 12/26 2021-05-12T08:59:07.372 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:17.0.0-4095.ga8c673d5.el8.x86_64 13/26 2021-05-12T08:59:07.547 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:17.0.0-4095.ga8c673d5.el8.x86_64 14/26 2021-05-12T08:59:07.740 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:17.0.0-4095.ga8c673d5.el8.x86_64 15/26 2021-05-12T08:59:07.918 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:17.0.0-4095.ga8c673d5.el8.x86_6 16/26 2021-05-12T08:59:07.996 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:17.0.0-4095.ga8c673d5.el8.x86_6 16/26 2021-05-12T08:59:08.174 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2021-05-12T08:59:08.328 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/26 2021-05-12T08:59:08.477 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2021-05-12T08:59:08.685 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:17.0.0-4095.ga8c673d5.el8.x8 20/26 2021-05-12T08:59:13.078 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:17.0.0-4095.ga8c673d5.el8.x86_64 21/26 2021-05-12T08:59:13.293 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:17.0.0-4095.ga8c673d5.el8.x86_64 21/26 2021-05-12T08:59:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:17.0.0-4095.ga8c673d5.el8.x86_64 21/26 2021-05-12T08:59:14.268 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:17.0.0-4095.ga8c673d5.el8.x86_64 22/26 2021-05-12T08:59:14.359 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:17.0.0-4095.ga8c673d5.el8.x86_64 22/26 2021-05-12T08:59:14.467 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:17.0.0-4095.ga8c673d5.el8.x86_64 23/26 2021-05-12T08:59:32.064 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:17.0.0-4095.ga8c673d5.el8.x86_64 23/26 2021-05-12T08:59:32.065 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2021-05-12T08:59:32.067 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2021-05-12T08:59:32.067 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /dev 2021-05-12T08:59:32.067 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /run 2021-05-12T08:59:32.068 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2021-05-12T08:59:32.069 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2021-05-12T08:59:32.069 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2021-05-12T08:59:32.070 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2021-05-12T08:59:32.072 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /dev 2021-05-12T08:59:32.072 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:59:33.324 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:17.0.0-4095.ga8c673d5.el8.x86_64 24/26 2021-05-12T08:59:33.389 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-4095.ga8c673d5.el8.x86_64 24/26 2021-05-12T08:59:33.389 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-05-12T08:59:33.559 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-05-12T08:59:33.559 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2021-05-12T08:59:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2021-05-12T08:59:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/26 2021-05-12T08:59:34.278 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:17.0.0-4095.ga8c673d5.el8.x86_64 2/26 2021-05-12T08:59:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:17.0.0-4095.ga8c673d5.el8.x86_64 3/26 2021-05-12T08:59:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:17.0.0-4095.ga8c673d5.el8.x86_64 4/26 2021-05-12T08:59:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:17.0.0-4095.ga8c673d5.el8.x86_64 5/26 2021-05-12T08:59:34.279 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:17.0.0-4095.ga8c673d5.el8.x86_6 6/26 2021-05-12T08:59:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:17.0.0-4095.ga8c673d5.el8.x86_64 7/26 2021-05-12T08:59:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:17.0.0-4095.ga8c673d5.el8. 8/26 2021-05-12T08:59:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:17.0.0-4095.ga8c673d5.el8.x8 9/26 2021-05-12T08:59:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:17.0.0-4095.ga8c673d5.el8.x86_64 10/26 2021-05-12T08:59:34.280 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:17.0.0-4095.ga8c673d5.el8.x86_64 11/26 2021-05-12T08:59:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:17.0.0-4095.ga8c673d5.el8.x86_64 12/26 2021-05-12T08:59:34.281 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:17.0.0-4095.ga8c673d5.el8.x86_64 13/26 2021-05-12T08:59:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/26 2021-05-12T08:59:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2021-05-12T08:59:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2021-05-12T08:59:34.282 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 17/26 2021-05-12T08:59:34.283 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2021-05-12T08:59:34.283 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 19/26 2021-05-12T08:59:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2021-05-12T08:59:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2021-05-12T08:59:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2021-05-12T08:59:34.286 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:17.0.0-4095.ga8c673d5.el8.x86_64 23/26 2021-05-12T08:59:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2021-05-12T08:59:34.287 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:17.0.0-4095.ga8c673d5.el8.x86_64 25/26 2021-05-12T08:59:35.113 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2021-05-12T08:59:35.115 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T08:59:35.413 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:59:35.414 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2021-05-12T08:59:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:59:35.415 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T08:59:35.415 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.416 INFO:teuthology.orchestra.run.smithi028.stdout: fmt-6.2.1-1.el8.x86_64 2021-05-12T08:59:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-05-12T08:59:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: leveldb-1.22-1.el8.x86_64 2021-05-12T08:59:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.417 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.2-3.el8.x86_64 2021-05-12T08:59:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-05-12T08:59:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-05-12T08:59:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.418 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.3.1-3.el8.x86_64 2021-05-12T08:59:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-05-12T08:59:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.419 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-05-12T08:59:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.420 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T08:59:35.421 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:59:35.421 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T08:59:36.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-test 2021-05-12T08:59:36.650 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:39 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T08:59:37.365 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T08:59:37.366 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T08:59:37.366 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2021-05-12T08:59:37.366 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T08:59:37.367 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T08:59:37.367 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 43 M 2021-05-12T08:59:37.367 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2021-05-12T08:59:37.367 INFO:teuthology.orchestra.run.smithi028.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-05-12T08:59:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 159 k 2021-05-12T08:59:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: libxslt x86_64 1.1.32-5.el8 CentOS-Base 250 k 2021-05-12T08:59:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-05-12T08:59:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-05-12T08:59:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-05-12T08:59:37.369 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:59:37.369 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T08:59:37.369 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T08:59:37.369 INFO:teuthology.orchestra.run.smithi028.stdout:Install 7 Packages 2021-05-12T08:59:37.369 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:59:37.370 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 44 M 2021-05-12T08:59:37.370 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 243 M 2021-05-12T08:59:37.371 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T08:59:37.434 INFO:teuthology.orchestra.run.smithi028.stdout:(1/7): jq-1.5-12.el8.x86_64.rpm 2.4 MB/s | 161 kB 00:00 2021-05-12T08:59:37.437 INFO:teuthology.orchestra.run.smithi028.stdout:(2/7): oniguruma-6.8.2-2.el8.x86_64.rpm 51 MB/s | 187 kB 00:00 2021-05-12T08:59:37.442 INFO:teuthology.orchestra.run.smithi028.stdout:(3/7): socat-1.7.3.3-2.el8.x86_64.rpm 67 MB/s | 302 kB 00:00 2021-05-12T08:59:37.456 INFO:teuthology.orchestra.run.smithi028.stdout:(4/7): libxslt-1.1.32-5.el8.x86_64.rpm 18 MB/s | 250 kB 00:00 2021-05-12T08:59:37.541 INFO:teuthology.orchestra.run.smithi028.stdout:(5/7): libcephsqlite-17.0.0-4095.ga8c673d5.el8. 926 kB/s | 159 kB 00:00 2021-05-12T08:59:37.547 INFO:teuthology.orchestra.run.smithi028.stdout:(6/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 756 kB/s | 69 kB 00:00 2021-05-12T08:59:39.366 INFO:teuthology.orchestra.run.smithi028.stdout:(7/7): ceph-test-17.0.0-4095.ga8c673d5.el8.x86_ 21 MB/s | 43 MB 00:01 2021-05-12T08:59:39.370 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T08:59:39.370 INFO:teuthology.orchestra.run.smithi028.stdout:Total 22 MB/s | 44 MB 00:02 2021-05-12T08:59:39.374 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T08:59:39.401 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T08:59:39.401 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T08:59:39.656 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T08:59:39.658 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T08:59:40.161 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T08:59:40.333 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 1/7 2021-05-12T08:59:40.494 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2021-05-12T08:59:40.635 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/7 2021-05-12T08:59:40.797 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-05-12T08:59:40.859 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/7 2021-05-12T08:59:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : jq-1.5-12.el8.x86_64 5/7 2021-05-12T08:59:41.483 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:17.0.0-4095.ga8c673d5.el8.x86_64 6/7 2021-05-12T08:59:46.717 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:17.0.0-4095.ga8c673d5.el8.x86_64 6/7 2021-05-12T08:59:46.919 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:17.0.0-4095.ga8c673d5.el8.x86_64 7/7 2021-05-12T08:59:47.166 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-test-2:17.0.0-4095.ga8c673d5.el8.x86_64 7/7 2021-05-12T08:59:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/7 2021-05-12T08:59:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:17.0.0-4095.ga8c673d5.el8.x86_64 2/7 2021-05-12T08:59:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : jq-1.5-12.el8.x86_64 3/7 2021-05-12T08:59:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-05-12T08:59:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 5/7 2021-05-12T08:59:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 6/7 2021-05-12T08:59:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 7/7 2021-05-12T08:59:47.581 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T08:59:48.292 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:59:48.293 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T08:59:48.293 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:17.0.0-4095.ga8c673d5.el8.x86_64 jq-1.5-12.el8.x86_64 2021-05-12T08:59:48.293 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:17.0.0-4095.ga8c673d5.el8.x86_64 libxslt-1.1.32-5.el8.x86_64 2021-05-12T08:59:48.293 INFO:teuthology.orchestra.run.smithi028.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.3.3-2.el8.x86_64 2021-05-12T08:59:48.293 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-05-12T08:59:48.294 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:59:48.294 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T08:59:48.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph 2021-05-12T08:59:49.136 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:52 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T08:59:49.843 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T08:59:49.847 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2021-05-12T08:59:49.848 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2021-05-12T08:59:49.848 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2021-05-12T08:59:49.848 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T08:59:49.849 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 6.4 k 2021-05-12T08:59:49.849 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2021-05-12T08:59:49.849 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 2.2 M 2021-05-12T08:59:49.849 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 1.5 M 2021-05-12T08:59:49.850 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:17.0.0-4095.ga8c673d5.el8 ceph-noarch 204 k 2021-05-12T08:59:49.850 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 3.8 M 2021-05-12T08:59:49.850 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 16 M 2021-05-12T08:59:49.850 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asn1crypto noarch 0.24.0-3.el8 CentOS-Base 181 k 2021-05-12T08:59:49.851 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-05-12T08:59:49.851 INFO:teuthology.orchestra.run.smithi028.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-05-12T08:59:49.854 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-05-12T08:59:49.854 INFO:teuthology.orchestra.run.smithi028.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-05-12T08:59:49.855 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-05-12T08:59:49.855 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-05-12T08:59:49.856 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 2.3-3.el8 CentOS-Base 510 k 2021-05-12T08:59:49.856 INFO:teuthology.orchestra.run.smithi028.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-05-12T08:59:49.857 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-05-12T08:59:49.857 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-05-12T08:59:49.858 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-05-12T08:59:49.858 INFO:teuthology.orchestra.run.smithi028.stdout: python3-lxml x86_64 4.2.3-1.el8 CentOS-AppStream 1.5 M 2021-05-12T08:59:49.858 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-05-12T08:59:49.859 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-05-12T08:59:49.859 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-05-12T08:59:49.859 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-05-12T08:59:49.859 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-05-12T08:59:49.860 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 18.0.0-1.el8 CentOS-AppStream 103 k 2021-05-12T08:59:49.860 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-05-12T08:59:49.860 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-05-12T08:59:49.860 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-05-12T08:59:49.861 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-05-12T08:59:49.861 INFO:teuthology.orchestra.run.smithi028.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-05-12T08:59:49.861 INFO:teuthology.orchestra.run.smithi028.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-05-12T08:59:49.861 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-05-12T08:59:49.862 INFO:teuthology.orchestra.run.smithi028.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-05-12T08:59:49.862 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.24.2-4.el8 CentOS-Base 176 k 2021-05-12T08:59:49.862 INFO:teuthology.orchestra.run.smithi028.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-05-12T08:59:49.862 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-05-12T08:59:49.863 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-05-12T08:59:49.863 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-05-12T08:59:49.863 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-05-12T08:59:49.863 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-05-12T08:59:49.865 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2021-05-12T08:59:49.866 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-05-12T08:59:49.866 INFO:teuthology.orchestra.run.smithi028.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-05-12T08:59:49.866 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:59:49.866 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T08:59:49.867 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2021-05-12T08:59:49.867 INFO:teuthology.orchestra.run.smithi028.stdout:Install 42 Packages 2021-05-12T08:59:49.867 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T08:59:49.868 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 30 M 2021-05-12T08:59:49.868 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 126 M 2021-05-12T08:59:49.869 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T08:59:49.979 INFO:teuthology.orchestra.run.smithi028.stdout:(1/42): ceph-17.0.0-4095.ga8c673d5.el8.x86_64.r 52 kB/s | 6.4 kB 00:00 2021-05-12T08:59:50.341 INFO:teuthology.orchestra.run.smithi028.stdout:(2/42): ceph-mds-17.0.0-4095.ga8c673d5.el8.x86_ 4.6 MB/s | 2.2 MB 00:00 2021-05-12T08:59:50.344 INFO:teuthology.orchestra.run.smithi028.stdout:(3/42): ceph-mgr-17.0.0-4095.ga8c673d5.el8.x86_ 3.0 MB/s | 1.5 MB 00:00 2021-05-12T08:59:50.368 INFO:teuthology.orchestra.run.smithi028.stdout:(4/42): ceph-mon-17.0.0-4095.ga8c673d5.el8.x86_ 9.8 MB/s | 3.8 MB 00:00 2021-05-12T08:59:50.395 INFO:teuthology.orchestra.run.smithi028.stdout:(5/42): ceph-mgr-modules-core-17.0.0-4095.ga8c6 4.0 MB/s | 204 kB 00:00 2021-05-12T08:59:50.404 INFO:teuthology.orchestra.run.smithi028.stdout:(6/42): python3-html5lib-0.999999999-6.el8.noar 5.8 MB/s | 214 kB 00:00 2021-05-12T08:59:50.407 INFO:teuthology.orchestra.run.smithi028.stdout:(7/42): python3-mako-1.0.6-13.el8.noarch.rpm 58 MB/s | 157 kB 00:00 2021-05-12T08:59:50.408 INFO:teuthology.orchestra.run.smithi028.stdout:(8/42): python3-markupsafe-0.23-19.el8.x86_64.r 27 MB/s | 39 kB 00:00 2021-05-12T08:59:50.411 INFO:teuthology.orchestra.run.smithi028.stdout:(9/42): python3-pyOpenSSL-18.0.0-1.el8.noarch.r 51 MB/s | 103 kB 00:00 2021-05-12T08:59:50.412 INFO:teuthology.orchestra.run.smithi028.stdout:(10/42): python3-pytz-2017.2-9.el8.noarch.rpm 39 MB/s | 54 kB 00:00 2021-05-12T08:59:50.413 INFO:teuthology.orchestra.run.smithi028.stdout:(11/42): python3-webencodings-0.5.1-6.el8.noarc 26 MB/s | 27 kB 00:00 2021-05-12T08:59:50.419 INFO:teuthology.orchestra.run.smithi028.stdout:(12/42): python3-werkzeug-0.12.2-4.el8.noarch.r 74 MB/s | 457 kB 00:00 2021-05-12T08:59:50.423 INFO:teuthology.orchestra.run.smithi028.stdout:(13/42): python3-asn1crypto-0.24.0-3.el8.noarch 50 MB/s | 181 kB 00:00 2021-05-12T08:59:50.427 INFO:teuthology.orchestra.run.smithi028.stdout:(14/42): python3-cffi-1.11.5-5.el8.x86_64.rpm 58 MB/s | 237 kB 00:00 2021-05-12T08:59:50.431 INFO:teuthology.orchestra.run.smithi028.stdout:(15/42): python3-chardet-3.0.4-7.el8.noarch.rpm 52 MB/s | 195 kB 00:00 2021-05-12T08:59:50.441 INFO:teuthology.orchestra.run.smithi028.stdout:(16/42): python3-cryptography-2.3-3.el8.x86_64. 64 MB/s | 510 kB 00:00 2021-05-12T08:59:50.444 INFO:teuthology.orchestra.run.smithi028.stdout:(17/42): python3-idna-2.5-5.el8.noarch.rpm 37 MB/s | 97 kB 00:00 2021-05-12T08:59:50.445 INFO:teuthology.orchestra.run.smithi028.stdout:(18/42): python3-pycparser-2.14-14.el8.noarch.r 42 MB/s | 109 kB 00:00 2021-05-12T08:59:50.446 INFO:teuthology.orchestra.run.smithi028.stdout:(19/42): python3-pysocks-1.6.8-3.el8.noarch.rpm 23 MB/s | 34 kB 00:00 2021-05-12T08:59:50.449 INFO:teuthology.orchestra.run.smithi028.stdout:(20/42): python3-requests-2.20.0-2.1.el8_1.noar 42 MB/s | 123 kB 00:00 2021-05-12T08:59:50.454 INFO:teuthology.orchestra.run.smithi028.stdout:(21/42): python3-urllib3-1.24.2-4.el8.noarch.rp 45 MB/s | 176 kB 00:00 2021-05-12T08:59:50.458 INFO:teuthology.orchestra.run.smithi028.stdout:(22/42): python3-lxml-4.2.3-1.el8.x86_64.rpm 24 MB/s | 1.5 MB 00:00 2021-05-12T08:59:50.547 INFO:teuthology.orchestra.run.smithi028.stdout:(23/42): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 471 kB/s | 44 kB 00:00 2021-05-12T08:59:50.568 INFO:teuthology.orchestra.run.smithi028.stdout:(24/42): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.6 MB/s | 185 kB 00:00 2021-05-12T08:59:50.612 INFO:teuthology.orchestra.run.smithi028.stdout:(25/42): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.6 MB/s | 173 kB 00:00 2021-05-12T08:59:50.614 INFO:teuthology.orchestra.run.smithi028.stdout:(26/42): python3-cherrypy-18.4.0-1.el8.noarch.r 8.2 MB/s | 384 kB 00:00 2021-05-12T08:59:50.635 INFO:teuthology.orchestra.run.smithi028.stdout:(27/42): python3-cssselect-0.9.2-10.el8.noarch. 1.7 MB/s | 40 kB 00:00 2021-05-12T08:59:50.637 INFO:teuthology.orchestra.run.smithi028.stdout:(28/42): python3-jaraco-6.2-6.el8.noarch.rpm 500 kB/s | 11 kB 00:00 2021-05-12T08:59:50.657 INFO:teuthology.orchestra.run.smithi028.stdout:(29/42): python3-jaraco-functools-2.0-4.el8.noa 792 kB/s | 18 kB 00:00 2021-05-12T08:59:50.660 INFO:teuthology.orchestra.run.smithi028.stdout:(30/42): python3-logutils-0.3.5-11.el8.noarch.r 2.1 MB/s | 49 kB 00:00 2021-05-12T08:59:50.681 INFO:teuthology.orchestra.run.smithi028.stdout:(31/42): python3-more-itertools-7.2.0-3.el8.noa 2.6 MB/s | 59 kB 00:00 2021-05-12T08:59:50.707 INFO:teuthology.orchestra.run.smithi028.stdout:(32/42): python3-portend-2.6-1.el8.noarch.rpm 703 kB/s | 16 kB 00:00 2021-05-12T08:59:50.707 INFO:teuthology.orchestra.run.smithi028.stdout:(33/42): python3-pecan-1.3.2-9.el8.noarch.rpm 6.2 MB/s | 283 kB 00:00 2021-05-12T08:59:50.726 INFO:teuthology.orchestra.run.smithi028.stdout:(34/42): python3-simplegeneric-0.8.1-17.el8.noa 839 kB/s | 19 kB 00:00 2021-05-12T08:59:50.726 INFO:teuthology.orchestra.run.smithi028.stdout:(35/42): python3-singledispatch-3.4.0.3-18.el8. 1.0 MB/s | 24 kB 00:00 2021-05-12T08:59:50.748 INFO:teuthology.orchestra.run.smithi028.stdout:(36/42): python3-tempora-1.14.1-5.el8.noarch.rp 1.3 MB/s | 29 kB 00:00 2021-05-12T08:59:50.751 INFO:teuthology.orchestra.run.smithi028.stdout:(37/42): python3-trustme-0.6.0-4.el8.noarch.rpm 1.2 MB/s | 27 kB 00:00 2021-05-12T08:59:50.776 INFO:teuthology.orchestra.run.smithi028.stdout:(38/42): python3-webob-1.8.5-1.el8.1.noarch.rpm 10 MB/s | 251 kB 00:00 2021-05-12T08:59:50.793 INFO:teuthology.orchestra.run.smithi028.stdout:(39/42): python3-waitress-1.2.1-2.el8.1.noarch. 5.3 MB/s | 241 kB 00:00 2021-05-12T08:59:50.798 INFO:teuthology.orchestra.run.smithi028.stdout:(40/42): python3-webtest-2.0.33-1.el8.noarch.rp 3.6 MB/s | 85 kB 00:00 2021-05-12T08:59:50.814 INFO:teuthology.orchestra.run.smithi028.stdout:(41/42): python3-zc-lockfile-2.0-2.el8.noarch.r 1.0 MB/s | 23 kB 00:00 2021-05-12T08:59:51.184 INFO:teuthology.orchestra.run.smithi028.stdout:(42/42): ceph-osd-17.0.0-4095.ga8c673d5.el8.x86 19 MB/s | 16 MB 00:00 2021-05-12T08:59:51.188 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T08:59:51.188 INFO:teuthology.orchestra.run.smithi028.stdout:Total 22 MB/s | 30 MB 00:01 2021-05-12T08:59:51.204 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T08:59:51.250 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T08:59:51.251 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T08:59:51.568 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T08:59:51.571 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T08:59:53.018 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T08:59:53.192 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/42 2021-05-12T08:59:53.409 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-idna-2.5-5.el8.noarch 2/42 2021-05-12T08:59:53.581 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 3/42 2021-05-12T08:59:53.825 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/42 2021-05-12T08:59:54.019 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/42 2021-05-12T08:59:54.167 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/42 2021-05-12T08:59:54.316 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/42 2021-05-12T08:59:54.457 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/42 2021-05-12T08:59:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/42 2021-05-12T08:59:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/42 2021-05-12T08:59:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 11/42 2021-05-12T08:59:55.101 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 12/42 2021-05-12T08:59:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.24.2-4.el8.noarch 13/42 2021-05-12T08:59:55.590 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 14/42 2021-05-12T08:59:55.783 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 15/42 2021-05-12T08:59:56.035 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 16/42 2021-05-12T08:59:56.292 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 17/42 2021-05-12T08:59:56.510 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 18/42 2021-05-12T08:59:56.845 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asn1crypto-0.24.0-3.el8.noarch 19/42 2021-05-12T08:59:57.092 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-2.3-3.el8.x86_64 20/42 2021-05-12T08:59:57.270 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-18.0.0-1.el8.noarch 21/42 2021-05-12T08:59:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 22/42 2021-05-12T08:59:57.772 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 23/42 2021-05-12T08:59:57.999 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/42 2021-05-12T08:59:58.250 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 25/42 2021-05-12T08:59:58.478 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 26/42 2021-05-12T08:59:59.118 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 27/42 2021-05-12T08:59:59.323 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 28/42 2021-05-12T08:59:59.502 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 29/42 2021-05-12T08:59:59.682 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 30/42 2021-05-12T08:59:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 31/42 2021-05-12T09:00:00.114 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-2.6-1.el8.noarch 32/42 2021-05-12T09:00:00.367 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 33/42 2021-05-12T09:00:00.596 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 34/42 2021-05-12T09:00:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 35/42 2021-05-12T09:00:01.241 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 36/42 2021-05-12T09:00:01.594 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-4095.ga8c673d5.el8. 37/42 2021-05-12T09:00:01.745 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:17.0.0-4095.ga8c673d5.el8.x86_64 38/42 2021-05-12T09:00:03.327 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:17.0.0-4095.ga8c673d5.el8.x86_64 38/42 2021-05-12T09:00:03.486 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:17.0.0-4095.ga8c673d5.el8.x86_64 39/42 2021-05-12T09:00:03.910 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:17.0.0-4095.ga8c673d5.el8.x86_64 39/42 2021-05-12T09:00:04.069 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:17.0.0-4095.ga8c673d5.el8.x86_64 40/42 2021-05-12T09:00:04.302 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:17.0.0-4095.ga8c673d5.el8.x86_64 40/42 2021-05-12T09:00:04.719 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:17.0.0-4095.ga8c673d5.el8.x86_64 41/42 2021-05-12T09:00:04.743 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:17.0.0-4095.ga8c673d5.el8.x86_64 41/42 2021-05-12T09:00:04.869 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:17.0.0-4095.ga8c673d5.el8.x86_64 42/42 2021-05-12T09:00:05.254 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-2:17.0.0-4095.ga8c673d5.el8.x86_64 42/42 2021-05-12T09:00:05.257 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/42 2021-05-12T09:00:05.257 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:17.0.0-4095.ga8c673d5.el8.x86_64 2/42 2021-05-12T09:00:05.257 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:17.0.0-4095.ga8c673d5.el8.x86_64 3/42 2021-05-12T09:00:05.257 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:17.0.0-4095.ga8c673d5.el8.x86_64 4/42 2021-05-12T09:00:05.258 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:17.0.0-4095.ga8c673d5.el8.x86_64 5/42 2021-05-12T09:00:05.258 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-4095.ga8c673d5.el8. 6/42 2021-05-12T09:00:05.259 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/42 2021-05-12T09:00:05.259 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/42 2021-05-12T09:00:05.260 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/42 2021-05-12T09:00:05.261 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/42 2021-05-12T09:00:05.261 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-18.0.0-1.el8.noarch 11/42 2021-05-12T09:00:05.262 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 12/42 2021-05-12T09:00:05.262 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 13/42 2021-05-12T09:00:05.262 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 14/42 2021-05-12T09:00:05.263 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asn1crypto-0.24.0-3.el8.noarch 15/42 2021-05-12T09:00:05.263 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 16/42 2021-05-12T09:00:05.264 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 17/42 2021-05-12T09:00:05.264 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-2.3-3.el8.x86_64 18/42 2021-05-12T09:00:05.264 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-idna-2.5-5.el8.noarch 19/42 2021-05-12T09:00:05.267 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 20/42 2021-05-12T09:00:05.267 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 21/42 2021-05-12T09:00:05.267 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 22/42 2021-05-12T09:00:05.268 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.24.2-4.el8.noarch 23/42 2021-05-12T09:00:05.268 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 24/42 2021-05-12T09:00:05.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 25/42 2021-05-12T09:00:05.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 26/42 2021-05-12T09:00:05.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 27/42 2021-05-12T09:00:05.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 28/42 2021-05-12T09:00:05.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 29/42 2021-05-12T09:00:05.275 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 30/42 2021-05-12T09:00:05.276 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 31/42 2021-05-12T09:00:05.276 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 32/42 2021-05-12T09:00:05.276 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 33/42 2021-05-12T09:00:05.276 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-2.6-1.el8.noarch 34/42 2021-05-12T09:00:05.276 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 35/42 2021-05-12T09:00:05.287 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 36/42 2021-05-12T09:00:05.287 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 37/42 2021-05-12T09:00:05.288 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 38/42 2021-05-12T09:00:05.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 39/42 2021-05-12T09:00:05.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 40/42 2021-05-12T09:00:05.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 41/42 2021-05-12T09:00:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 42/42 2021-05-12T09:00:06.149 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T09:00:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:06.381 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T09:00:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T09:00:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T09:00:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T09:00:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:17.0.0-4095.ga8c673d5.el8.noarch 2021-05-12T09:00:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T09:00:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T09:00:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asn1crypto-0.24.0-3.el8.noarch 2021-05-12T09:00:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-05-12T09:00:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-05-12T09:00:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-05-12T09:00:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-05-12T09:00:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-05-12T09:00:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-05-12T09:00:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-2.3-3.el8.x86_64 2021-05-12T09:00:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-05-12T09:00:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-05-12T09:00:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: python3-idna-2.5-5.el8.noarch 2021-05-12T09:00:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-6.2-6.el8.noarch 2021-05-12T09:00:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-05-12T09:00:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-05-12T09:00:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-05-12T09:00:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.0.6-13.el8.noarch 2021-05-12T09:00:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-05-12T09:00:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-05-12T09:00:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-05-12T09:00:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-2.6-1.el8.noarch 2021-05-12T09:00:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-18.0.0-1.el8.noarch 2021-05-12T09:00:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.14-14.el8.noarch 2021-05-12T09:00:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-05-12T09:00:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2017.2-9.el8.noarch 2021-05-12T09:00:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-05-12T09:00:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-05-12T09:00:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-05-12T09:00:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-05-12T09:00:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-05-12T09:00:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.24.2-4.el8.noarch 2021-05-12T09:00:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-05-12T09:00:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-05-12T09:00:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-05-12T09:00:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-05-12T09:00:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-05-12T09:00:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-05-12T09:00:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:06.394 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:06.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-base 2021-05-12T09:00:07.167 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:10 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:00:07.683 INFO:teuthology.orchestra.run.smithi028.stdout:Package ceph-base-2:17.0.0-4095.ga8c673d5.el8.x86_64 is already installed. 2021-05-12T09:00:07.858 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:00:07.859 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2021-05-12T09:00:07.859 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:07.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install cephadm 2021-05-12T09:00:08.499 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:11 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:00:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:00:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:09.188 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2021-05-12T09:00:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T09:00:09.189 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:17.0.0-4095.ga8c673d5.el8 ceph-noarch 67 k 2021-05-12T09:00:09.189 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T09:00:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2021-05-12T09:00:09.190 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 67 k 2021-05-12T09:00:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 296 k 2021-05-12T09:00:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T09:00:09.375 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm-17.0.0-4095.ga8c673d5.el8.noarch.rpm 362 kB/s | 67 kB 00:00 2021-05-12T09:00:09.376 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T09:00:09.376 INFO:teuthology.orchestra.run.smithi028.stdout:Total 360 kB/s | 67 kB 00:00 2021-05-12T09:00:09.377 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T09:00:09.383 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T09:00:09.383 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T09:00:09.385 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T09:00:09.391 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T09:00:09.437 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T09:00:11.547 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:17.0.0-4095.ga8c673d5.el8.noarch 1/1 2021-05-12T09:00:11.765 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:17.0.0-4095.ga8c673d5.el8.noarch 1/1 2021-05-12T09:00:11.946 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:17.0.0-4095.ga8c673d5.el8.noarch 1/1 2021-05-12T09:00:12.205 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:17.0.0-4095.ga8c673d5.el8.noarch 1/1 2021-05-12T09:00:12.205 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T09:00:12.465 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:12.466 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T09:00:12.467 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:17.0.0-4095.ga8c673d5.el8.noarch 2021-05-12T09:00:12.467 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:12.468 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:12.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-immutable-object-cache 2021-05-12T09:00:13.204 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:16 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:00:13.937 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:00:13.938 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2021-05-12T09:00:13.939 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:13.939 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T09:00:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 167 k 2021-05-12T09:00:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:13.940 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T09:00:13.940 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:13.940 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2021-05-12T09:00:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:13.941 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 167 k 2021-05-12T09:00:13.941 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 633 k 2021-05-12T09:00:13.941 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T09:00:14.109 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-immutable-object-cache-17.0.0-4095.ga8c673 985 kB/s | 167 kB 00:00 2021-05-12T09:00:14.110 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T09:00:14.110 INFO:teuthology.orchestra.run.smithi028.stdout:Total 978 kB/s | 167 kB 00:00 2021-05-12T09:00:14.111 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T09:00:14.124 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T09:00:14.125 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T09:00:14.179 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T09:00:14.182 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T09:00:14.299 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T09:00:14.543 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-4095.ga8c673d5. 1/1 2021-05-12T09:00:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-4095.ga8c673d5. 1/1 2021-05-12T09:00:15.311 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-4095.ga8c673d5. 1/1 2021-05-12T09:00:15.311 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T09:00:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:16.110 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T09:00:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T09:00:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:16.111 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:17.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr 2021-05-12T09:00:17.868 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:20 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:00:18.393 INFO:teuthology.orchestra.run.smithi028.stdout:Package ceph-mgr-2:17.0.0-4095.ga8c673d5.el8.x86_64 is already installed. 2021-05-12T09:00:18.575 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:00:18.576 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2021-05-12T09:00:18.576 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:18.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-dashboard 2021-05-12T09:00:19.201 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:22 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:00:19.909 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:00:19.910 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:19.910 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2021-05-12T09:00:19.911 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:19.911 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T09:00:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:17.0.0-4095.ga8c673d5.el8 ceph-noarch 2.3 M 2021-05-12T09:00:19.912 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2021-05-12T09:00:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:17.0.0-4095.ga8c673d5.el8 ceph-noarch 24 k 2021-05-12T09:00:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:17.0.0-4095.ga8c673d5.el8 ceph-noarch 8.9 k 2021-05-12T09:00:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-05-12T09:00:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-05-12T09:00:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-05-12T09:00:19.913 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:19.913 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T09:00:19.914 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:19.914 INFO:teuthology.orchestra.run.smithi028.stdout:Install 6 Packages 2021-05-12T09:00:19.915 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:19.916 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 2.6 M 2021-05-12T09:00:19.916 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 15 M 2021-05-12T09:00:19.916 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T09:00:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:(1/6): ceph-prometheus-alerts-17.0.0-4095.ga8c6 81 kB/s | 8.9 kB 00:00 2021-05-12T09:00:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:(2/6): ceph-grafana-dashboards-17.0.0-4095.ga8c 197 kB/s | 24 kB 00:00 2021-05-12T09:00:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2021-05-12T09:00:20.117 INFO:teuthology.orchestra.run.smithi028.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 411 kB/s | 34 kB 00:00 2021-05-12T09:00:20.165 INFO:teuthology.orchestra.run.smithi028.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.7 MB/s | 196 kB 00:00 2021-05-12T09:00:20.208 INFO:teuthology.orchestra.run.smithi028.stdout:(6/6): ceph-mgr-dashboard-17.0.0-4095.ga8c673d5 7.7 MB/s | 2.3 MB 00:00 2021-05-12T09:00:20.210 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T09:00:20.211 INFO:teuthology.orchestra.run.smithi028.stdout:Total 8.6 MB/s | 2.6 MB 00:00 2021-05-12T09:00:20.212 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T09:00:20.223 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T09:00:20.223 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T09:00:20.350 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T09:00:20.353 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T09:00:20.633 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T09:00:20.839 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-05-12T09:00:21.029 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-05-12T09:00:21.257 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-05-12T09:00:21.383 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-4095.ga8c673d5.el8.n 4/6 2021-05-12T09:00:22.469 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-4095.ga8c673d5.el8. 5/6 2021-05-12T09:00:22.767 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-4095.ga8c673d5.el8.noarc 6/6 2021-05-12T09:00:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-4095.ga8c673d5.el8.noarc 6/6 2021-05-12T09:00:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-4095.ga8c673d5.el8. 1/6 2021-05-12T09:00:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-4095.ga8c673d5.el8.noarc 2/6 2021-05-12T09:00:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-4095.ga8c673d5.el8.n 3/6 2021-05-12T09:00:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-05-12T09:00:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-05-12T09:00:23.279 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-05-12T09:00:23.279 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T09:00:23.498 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:23.498 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T09:00:23.499 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:17.0.0-4095.ga8c673d5.el8.noarch 2021-05-12T09:00:23.499 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:17.0.0-4095.ga8c673d5.el8.noarch 2021-05-12T09:00:23.500 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:17.0.0-4095.ga8c673d5.el8.noarch 2021-05-12T09:00:23.500 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-05-12T09:00:23.500 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-05-12T09:00:23.501 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.4.1-12.el8.noarch 2021-05-12T09:00:23.501 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:23.501 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:23.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-diskprediction-local 2021-05-12T09:00:24.143 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:27 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:00:24.871 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:00:24.873 INFO:teuthology.orchestra.run.smithi028.stdout:===================================================================================================== 2021-05-12T09:00:24.873 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2021-05-12T09:00:24.873 INFO:teuthology.orchestra.run.smithi028.stdout:===================================================================================================== 2021-05-12T09:00:24.873 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T09:00:24.874 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-4095.ga8c673d5.el8 ceph-noarch 7.4 M 2021-05-12T09:00:24.874 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2021-05-12T09:00:24.874 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 8.3.1-5.1.el8 CentOS-Base 639 k 2021-05-12T09:00:24.874 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 8.3.1-5.1.el8 CentOS-Base 169 k 2021-05-12T09:00:24.874 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-05-12T09:00:24.875 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-threads x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-05-12T09:00:24.875 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-05-12T09:00:24.875 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-05-12T09:00:24.875 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.0.0-20.module_el8.3.0+562+e162826a CentOS-AppStream 15 M 2021-05-12T09:00:24.876 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:24.876 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T09:00:24.876 INFO:teuthology.orchestra.run.smithi028.stdout:===================================================================================================== 2021-05-12T09:00:24.877 INFO:teuthology.orchestra.run.smithi028.stdout:Install 8 Packages 2021-05-12T09:00:24.877 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:24.878 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 35 M 2021-05-12T09:00:24.878 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 194 M 2021-05-12T09:00:24.878 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T09:00:25.066 INFO:teuthology.orchestra.run.smithi028.stdout:(1/8): openblas-0.3.3-5.el8.x86_64.rpm 22 MB/s | 4.3 MB 00:00 2021-05-12T09:00:25.191 INFO:teuthology.orchestra.run.smithi028.stdout:(2/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 14 MB/s | 4.3 MB 00:00 2021-05-12T09:00:25.203 INFO:teuthology.orchestra.run.smithi028.stdout:(3/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 27 MB/s | 3.7 MB 00:00 2021-05-12T09:00:25.206 INFO:teuthology.orchestra.run.smithi028.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 14 MB/s | 225 kB 00:00 2021-05-12T09:00:25.243 INFO:teuthology.orchestra.run.smithi028.stdout:(5/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 17 MB/s | 639 kB 00:00 2021-05-12T09:00:25.256 INFO:teuthology.orchestra.run.smithi028.stdout:(6/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 13 MB/s | 169 kB 00:00 2021-05-12T09:00:25.504 INFO:teuthology.orchestra.run.smithi028.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.3.0+56 48 MB/s | 15 MB 00:00 2021-05-12T09:00:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-409 6.5 MB/s | 7.4 MB 00:01 2021-05-12T09:00:26.024 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T09:00:26.025 INFO:teuthology.orchestra.run.smithi028.stdout:Total 31 MB/s | 35 MB 00:01 2021-05-12T09:00:26.034 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T09:00:26.055 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T09:00:26.056 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T09:00:26.376 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T09:00:26.379 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T09:00:26.924 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T09:00:27.083 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-05-12T09:00:27.216 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-05-12T09:00:27.337 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-05-12T09:00:27.871 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-05-12T09:00:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.3-5.el8.x86_64 3/8 2021-05-12T09:00:28.688 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 3/8 2021-05-12T09:00:28.857 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-05-12T09:00:29.629 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-05-12T09:00:29.919 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-05-12T09:00:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-05-12T09:00:32.209 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-05-12T09:00:33.529 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 7/8 2021-05-12T09:00:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-4095.ga8c673d 8/8 2021-05-12T09:00:34.250 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-4095.ga8c673d 8/8 2021-05-12T09:00:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-4095.ga8c673d 1/8 2021-05-12T09:00:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 2/8 2021-05-12T09:00:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-05-12T09:00:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-05-12T09:00:34.251 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-05-12T09:00:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 6/8 2021-05-12T09:00:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 7/8 2021-05-12T09:00:34.545 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 8/8 2021-05-12T09:00:34.546 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T09:00:34.764 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:34.764 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T09:00:34.764 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:17.0.0-4095.ga8c673d5.el8.noarch 2021-05-12T09:00:34.765 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-05-12T09:00:34.765 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-05-12T09:00:34.765 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.3-5.el8.x86_64 2021-05-12T09:00:34.766 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-05-12T09:00:34.766 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-05-12T09:00:34.766 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-05-12T09:00:34.766 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x86_64 2021-05-12T09:00:34.767 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:34.767 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:34.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-rook 2021-05-12T09:00:35.415 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:38 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:00:36.119 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:00:36.121 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2021-05-12T09:00:36.121 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2021-05-12T09:00:36.122 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2021-05-12T09:00:36.122 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T09:00:36.123 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:17.0.0-4095.ga8c673d5.el8 ceph-noarch 26 k 2021-05-12T09:00:36.123 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2021-05-12T09:00:36.124 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-05-12T09:00:36.124 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-05-12T09:00:36.124 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-05-12T09:00:36.125 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-05-12T09:00:36.125 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-05-12T09:00:36.126 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-05-12T09:00:36.126 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-05-12T09:00:36.127 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-05-12T09:00:36.127 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-05-12T09:00:36.128 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-05-12T09:00:36.128 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-05-12T09:00:36.128 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-05-12T09:00:36.129 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:36.129 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T09:00:36.130 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2021-05-12T09:00:36.130 INFO:teuthology.orchestra.run.smithi028.stdout:Install 13 Packages 2021-05-12T09:00:36.131 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:36.132 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 1.9 M 2021-05-12T09:00:36.133 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 24 M 2021-05-12T09:00:36.133 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T09:00:36.158 INFO:teuthology.orchestra.run.smithi028.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 806 kB/s | 27 kB 00:00 2021-05-12T09:00:36.159 INFO:teuthology.orchestra.run.smithi028.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 575 kB/s | 20 kB 00:00 2021-05-12T09:00:36.161 INFO:teuthology.orchestra.run.smithi028.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 37 MB/s | 126 kB 00:00 2021-05-12T09:00:36.162 INFO:teuthology.orchestra.run.smithi028.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 38 MB/s | 110 kB 00:00 2021-05-12T09:00:36.165 INFO:teuthology.orchestra.run.smithi028.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 27 MB/s | 43 kB 00:00 2021-05-12T09:00:36.165 INFO:teuthology.orchestra.run.smithi028.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 42 MB/s | 155 kB 00:00 2021-05-12T09:00:36.232 INFO:teuthology.orchestra.run.smithi028.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 245 kB/s | 16 kB 00:00 2021-05-12T09:00:36.232 INFO:teuthology.orchestra.run.smithi028.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 488 kB/s | 33 kB 00:00 2021-05-12T09:00:36.247 INFO:teuthology.orchestra.run.smithi028.stdout:(9/13): ceph-mgr-rook-17.0.0-4095.ga8c673d5.el8 212 kB/s | 26 kB 00:00 2021-05-12T09:00:36.276 INFO:teuthology.orchestra.run.smithi028.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 1.8 MB/s | 81 kB 00:00 2021-05-12T09:00:36.299 INFO:teuthology.orchestra.run.smithi028.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 2.6 MB/s | 61 kB 00:00 2021-05-12T09:00:36.357 INFO:teuthology.orchestra.run.smithi028.stdout:(12/13): python3-rsa-3.4.2-16.el8.noarch.rpm 672 kB/s | 73 kB 00:00 2021-05-12T09:00:36.431 INFO:teuthology.orchestra.run.smithi028.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.9 MB/s | 1.2 MB 00:00 2021-05-12T09:00:36.432 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T09:00:36.433 INFO:teuthology.orchestra.run.smithi028.stdout:Total 6.2 MB/s | 1.9 MB 00:00 2021-05-12T09:00:36.439 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T09:00:36.455 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T09:00:36.455 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T09:00:36.534 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T09:00:36.537 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T09:00:37.014 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T09:00:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-05-12T09:00:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-05-12T09:00:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-05-12T09:00:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-05-12T09:00:38.152 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-05-12T09:00:38.398 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-05-12T09:00:38.701 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-05-12T09:00:38.919 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-05-12T09:00:40.262 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-05-12T09:00:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-05-12T09:00:40.707 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-05-12T09:00:40.908 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-05-12T09:00:41.081 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:17.0.0-4095.ga8c673d5.el8.noarch 13/13 2021-05-12T09:00:41.338 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-4095.ga8c673d5.el8.noarch 13/13 2021-05-12T09:00:41.339 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:17.0.0-4095.ga8c673d5.el8.noarch 1/13 2021-05-12T09:00:41.339 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-05-12T09:00:41.340 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-05-12T09:00:41.340 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-05-12T09:00:41.341 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-05-12T09:00:41.341 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-05-12T09:00:41.341 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-05-12T09:00:41.342 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-05-12T09:00:41.342 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-05-12T09:00:41.343 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-05-12T09:00:41.343 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-05-12T09:00:41.344 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-05-12T09:00:41.667 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-05-12T09:00:41.667 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T09:00:41.888 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:41.888 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T09:00:41.889 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:17.0.0-4095.ga8c673d5.el8.noarch 2021-05-12T09:00:41.889 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-05-12T09:00:41.889 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-05-12T09:00:41.889 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-05-12T09:00:41.890 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-05-12T09:00:41.890 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-05-12T09:00:41.890 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-05-12T09:00:41.890 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-05-12T09:00:41.890 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-05-12T09:00:41.891 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-05-12T09:00:41.891 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-05-12T09:00:41.891 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-05-12T09:00:41.891 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-05-12T09:00:41.891 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:41.892 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:41.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-cephadm 2021-05-12T09:00:42.561 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:45 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:00:43.260 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:00:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:43.262 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2021-05-12T09:00:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T09:00:43.263 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:17.0.0-4095.ga8c673d5.el8 ceph-noarch 84 k 2021-05-12T09:00:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2021-05-12T09:00:43.263 INFO:teuthology.orchestra.run.smithi028.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-05-12T09:00:43.263 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-05-12T09:00:43.264 INFO:teuthology.orchestra.run.smithi028.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-05-12T09:00:43.264 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-05-12T09:00:43.264 INFO:teuthology.orchestra.run.smithi028.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-05-12T09:00:43.264 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:43.265 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T09:00:43.265 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:43.265 INFO:teuthology.orchestra.run.smithi028.stdout:Install 6 Packages 2021-05-12T09:00:43.265 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:43.266 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 5.8 M 2021-05-12T09:00:43.266 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 24 M 2021-05-12T09:00:43.267 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T09:00:43.301 INFO:teuthology.orchestra.run.smithi028.stdout:(1/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 14 MB/s | 538 kB 00:00 2021-05-12T09:00:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:(2/6): python3-babel-2.5.1-5.el8.noarch.rpm 48 MB/s | 4.8 MB 00:00 2021-05-12T09:00:43.368 INFO:teuthology.orchestra.run.smithi028.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 316 kB/s | 21 kB 00:00 2021-05-12T09:00:43.412 INFO:teuthology.orchestra.run.smithi028.stdout:(4/6): python3-remoto-1.1.4-4.el8.noarch.rpm 1.1 MB/s | 48 kB 00:00 2021-05-12T09:00:43.496 INFO:teuthology.orchestra.run.smithi028.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.6 MB/s | 355 kB 00:00 2021-05-12T09:00:43.590 INFO:teuthology.orchestra.run.smithi028.stdout:(6/6): ceph-mgr-cephadm-17.0.0-4095.ga8c673d5.e 257 kB/s | 84 kB 00:00 2021-05-12T09:00:43.592 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T09:00:43.593 INFO:teuthology.orchestra.run.smithi028.stdout:Total 18 MB/s | 5.8 MB 00:00 2021-05-12T09:00:43.595 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T09:00:43.606 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T09:00:43.606 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T09:00:43.691 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T09:00:43.695 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T09:00:43.979 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T09:00:44.223 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-05-12T09:00:44.479 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-05-12T09:00:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2021-05-12T09:00:46.127 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 4/6 2021-05-12T09:00:46.391 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-05-12T09:00:46.607 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-4095.ga8c673d5.el8.noarch 6/6 2021-05-12T09:00:46.860 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-4095.ga8c673d5.el8.noarch 6/6 2021-05-12T09:00:46.860 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-4095.ga8c673d5.el8.noarch 1/6 2021-05-12T09:00:46.860 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/6 2021-05-12T09:00:46.863 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/6 2021-05-12T09:00:46.863 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-05-12T09:00:46.864 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-05-12T09:00:47.133 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2021-05-12T09:00:47.134 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T09:00:47.371 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:47.372 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T09:00:47.372 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:17.0.0-4095.ga8c673d5.el8.noarch 2021-05-12T09:00:47.372 INFO:teuthology.orchestra.run.smithi028.stdout: python3-apipkg-1.5-6.el8.noarch 2021-05-12T09:00:47.372 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.5.1-5.el8.noarch 2021-05-12T09:00:47.373 INFO:teuthology.orchestra.run.smithi028.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-05-12T09:00:47.373 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-05-12T09:00:47.373 INFO:teuthology.orchestra.run.smithi028.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-05-12T09:00:47.373 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:47.373 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:47.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-fuse 2021-05-12T09:00:48.039 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:51 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:00:48.743 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:00:48.743 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:48.743 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2021-05-12T09:00:48.744 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:48.744 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T09:00:48.744 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 837 k 2021-05-12T09:00:48.744 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2021-05-12T09:00:48.744 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-05-12T09:00:48.745 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:48.745 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T09:00:48.745 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:48.745 INFO:teuthology.orchestra.run.smithi028.stdout:Install 2 Packages 2021-05-12T09:00:48.746 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:48.746 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 919 k 2021-05-12T09:00:48.747 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 3.3 M 2021-05-12T09:00:48.747 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T09:00:48.780 INFO:teuthology.orchestra.run.smithi028.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.3 MB/s | 82 kB 00:00 2021-05-12T09:00:49.050 INFO:teuthology.orchestra.run.smithi028.stdout:(2/2): ceph-fuse-17.0.0-4095.ga8c673d5.el8.x86_ 2.7 MB/s | 837 kB 00:00 2021-05-12T09:00:49.050 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T09:00:49.051 INFO:teuthology.orchestra.run.smithi028.stdout:Total 3.0 MB/s | 919 kB 00:00 2021-05-12T09:00:49.051 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T09:00:49.070 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T09:00:49.071 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T09:00:49.151 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T09:00:49.154 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T09:00:49.320 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T09:00:49.604 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-05-12T09:00:49.796 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:17.0.0-4095.ga8c673d5.el8.x86_64 2/2 2021-05-12T09:00:50.157 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-fuse-2:17.0.0-4095.ga8c673d5.el8.x86_64 2/2 2021-05-12T09:00:50.157 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/2 2021-05-12T09:00:50.403 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-05-12T09:00:50.403 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T09:00:50.652 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:50.652 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T09:00:50.652 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:17.0.0-4095.ga8c673d5.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-05-12T09:00:50.653 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:50.653 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:50.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librados-devel 2021-05-12T09:00:51.404 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:54 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:00:52.098 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:00:52.099 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:52.099 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2021-05-12T09:00:52.099 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:52.100 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T09:00:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 125 k 2021-05-12T09:00:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:52.100 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T09:00:52.100 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:52.101 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2021-05-12T09:00:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:52.101 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 125 k 2021-05-12T09:00:52.101 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 439 k 2021-05-12T09:00:52.101 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T09:00:52.267 INFO:teuthology.orchestra.run.smithi028.stdout:librados-devel-17.0.0-4095.ga8c673d5.el8.x86_64 742 kB/s | 125 kB 00:00 2021-05-12T09:00:52.269 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T09:00:52.269 INFO:teuthology.orchestra.run.smithi028.stdout:Total 736 kB/s | 125 kB 00:00 2021-05-12T09:00:52.269 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T09:00:52.276 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T09:00:52.276 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T09:00:52.331 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T09:00:52.570 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T09:00:52.570 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T09:00:52.623 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:00:52.856 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados-devel-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:00:53.099 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:00:53.100 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T09:00:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:53.353 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T09:00:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T09:00:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:53.353 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:53.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install libcephfs2 2021-05-12T09:00:54.094 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:57 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:00:54.647 INFO:teuthology.orchestra.run.smithi028.stdout:Package libcephfs2-2:17.0.0-4095.ga8c673d5.el8.x86_64 is already installed. 2021-05-12T09:00:54.823 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:00:54.824 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2021-05-12T09:00:54.824 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:54.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install libcephfs-devel 2021-05-12T09:00:55.455 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:58 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:00:56.174 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:00:56.175 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:56.175 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2021-05-12T09:00:56.175 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:56.175 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T09:00:56.176 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 24 k 2021-05-12T09:00:56.176 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:56.176 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T09:00:56.176 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:00:56.177 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2021-05-12T09:00:56.177 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:56.177 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 24 k 2021-05-12T09:00:56.177 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 100 k 2021-05-12T09:00:56.178 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T09:00:56.299 INFO:teuthology.orchestra.run.smithi028.stdout:libcephfs-devel-17.0.0-4095.ga8c673d5.el8.x86_6 192 kB/s | 24 kB 00:00 2021-05-12T09:00:56.300 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T09:00:56.300 INFO:teuthology.orchestra.run.smithi028.stdout:Total 190 kB/s | 24 kB 00:00 2021-05-12T09:00:56.300 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T09:00:56.305 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T09:00:56.305 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T09:00:56.308 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T09:00:56.310 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T09:00:56.369 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T09:00:56.545 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:00:56.769 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:00:57.009 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:00:57.010 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T09:00:57.224 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:57.224 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T09:00:57.224 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T09:00:57.225 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:00:57.225 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:57.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librados2 2021-05-12T09:00:57.871 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:00 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:00:58.405 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:17.0.0-4095.ga8c673d5.el8.x86_64 is already installed. 2021-05-12T09:00:58.581 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:00:58.581 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2021-05-12T09:00:58.581 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:58.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librbd1 2021-05-12T09:00:59.223 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:02 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:00:59.763 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:17.0.0-4095.ga8c673d5.el8.x86_64 is already installed. 2021-05-12T09:00:59.939 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:00:59.940 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2021-05-12T09:00:59.941 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:00:59.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rados 2021-05-12T09:01:00.581 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:03 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:01:01.419 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rados-2:17.0.0-4095.ga8c673d5.el8.x86_64 is already installed. 2021-05-12T09:01:01.595 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:01:01.597 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2021-05-12T09:01:01.597 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:01:01.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rgw 2021-05-12T09:01:02.505 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:05 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:01:03.029 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rgw-2:17.0.0-4095.ga8c673d5.el8.x86_64 is already installed. 2021-05-12T09:01:03.205 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:01:03.206 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2021-05-12T09:01:03.207 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:01:03.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-cephfs 2021-05-12T09:01:03.837 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:06 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:01:04.371 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-cephfs-2:17.0.0-4095.ga8c673d5.el8.x86_64 is already installed. 2021-05-12T09:01:04.554 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:01:04.555 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2021-05-12T09:01:04.555 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:01:04.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rbd 2021-05-12T09:01:05.201 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:08 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:01:05.729 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rbd-2:17.0.0-4095.ga8c673d5.el8.x86_64 is already installed. 2021-05-12T09:01:05.917 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:01:05.918 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2021-05-12T09:01:05.918 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:01:05.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-fuse 2021-05-12T09:01:06.547 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:09 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:01:07.255 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:01:07.255 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:01:07.256 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2021-05-12T09:01:07.256 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:01:07.257 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T09:01:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 90 k 2021-05-12T09:01:07.257 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:01:07.258 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T09:01:07.258 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:01:07.259 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2021-05-12T09:01:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:01:07.259 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 90 k 2021-05-12T09:01:07.260 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 320 k 2021-05-12T09:01:07.260 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T09:01:07.470 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-fuse-17.0.0-4095.ga8c673d5.el8.x86_64.rpm 418 kB/s | 90 kB 00:00 2021-05-12T09:01:07.471 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T09:01:07.472 INFO:teuthology.orchestra.run.smithi028.stdout:Total 415 kB/s | 90 kB 00:00 2021-05-12T09:01:07.472 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T09:01:07.485 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T09:01:07.486 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T09:01:07.542 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T09:01:07.545 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T09:01:07.764 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T09:01:08.602 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:01:08.859 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-fuse-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:01:09.126 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:01:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T09:01:09.384 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:01:09.384 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T09:01:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T09:01:09.385 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:01:09.385 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:01:09.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-mirror 2021-05-12T09:01:10.112 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:13 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:01:10.814 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:01:10.815 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:01:10.815 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2021-05-12T09:01:10.816 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:01:10.816 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T09:01:10.816 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 3.7 M 2021-05-12T09:01:10.816 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:01:10.817 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T09:01:10.817 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:01:10.817 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2021-05-12T09:01:10.817 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:01:10.818 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 3.7 M 2021-05-12T09:01:10.818 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 21 M 2021-05-12T09:01:10.818 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T09:01:12.358 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-mirror-17.0.0-4095.ga8c673d5.el8.x86_64.rpm 2.4 MB/s | 3.7 MB 00:01 2021-05-12T09:01:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T09:01:12.360 INFO:teuthology.orchestra.run.smithi028.stdout:Total 2.4 MB/s | 3.7 MB 00:01 2021-05-12T09:01:12.361 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T09:01:12.377 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T09:01:12.378 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T09:01:12.451 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T09:01:12.454 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T09:01:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T09:01:13.177 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:01:13.559 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:01:13.804 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:01:13.804 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T09:01:14.184 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:01:14.185 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T09:01:14.185 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T09:01:14.185 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:01:14.185 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:01:14.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-nbd 2021-05-12T09:01:15.040 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:02:18 ago on Wed 12 May 2021 08:58:57 AM UTC. 2021-05-12T09:01:15.779 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2021-05-12T09:01:15.780 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:01:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2021-05-12T09:01:15.781 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:01:15.782 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2021-05-12T09:01:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:17.0.0-4095.ga8c673d5.el8 ceph 170 k 2021-05-12T09:01:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:01:15.783 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2021-05-12T09:01:15.784 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2021-05-12T09:01:15.784 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2021-05-12T09:01:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:01:15.785 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 170 k 2021-05-12T09:01:15.785 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 671 k 2021-05-12T09:01:15.786 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2021-05-12T09:01:16.002 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-nbd-17.0.0-4095.ga8c673d5.el8.x86_64.rpm 769 kB/s | 170 kB 00:00 2021-05-12T09:01:16.003 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2021-05-12T09:01:16.003 INFO:teuthology.orchestra.run.smithi028.stdout:Total 764 kB/s | 170 kB 00:00 2021-05-12T09:01:16.003 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2021-05-12T09:01:16.018 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2021-05-12T09:01:16.018 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2021-05-12T09:01:16.075 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2021-05-12T09:01:16.078 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2021-05-12T09:01:16.202 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2021-05-12T09:01:16.392 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:01:16.637 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-nbd-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:01:16.905 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:17.0.0-4095.ga8c673d5.el8.x86_64 1/1 2021-05-12T09:01:16.905 INFO:teuthology.orchestra.run.smithi028.stdout:Installed products updated. 2021-05-12T09:01:17.141 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:01:17.142 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2021-05-12T09:01:17.142 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:17.0.0-4095.ga8c673d5.el8.x86_64 2021-05-12T09:01:17.142 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-12T09:01:17.142 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2021-05-12T09:01:17.322 DEBUG:teuthology.parallel:result is None 2021-05-12T09:01:17.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo lsb_release -is 2021-05-12T09:01:17.358 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS 2021-05-12T09:01:17.359 DEBUG:teuthology.misc:System to be installed: CentOS 2021-05-12T09:01:17.360 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a8c673d555488beec9ccda5812ba822e69c12ea9 2021-05-12T09:01:17.530 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-05-12T09:01:17.551 INFO:teuthology.orchestra.run.smithi028.stdout:17.0.0-4095.ga8c673d5.el8 2021-05-12T09:01:17.552 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-4095.ga8c673d5.el8 2021-05-12T09:01:17.552 INFO:teuthology.task.install:The correct ceph version 17.0.0-4095.ga8c673d5 is installed. 2021-05-12T09:01:17.553 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-05-12T09:01:17.553 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2021-05-12T09:01:17.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-05-12T09:01:17.615 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-05-12T09:01:17.616 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2021-05-12T09:01:17.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2021-05-12T09:01:17.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-05-12T09:01:17.745 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-05-12T09:01:17.745 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2021-05-12T09:01:17.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2021-05-12T09:01:17.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-05-12T09:01:17.875 INFO:teuthology.run_tasks:Running task workunit... 2021-05-12T09:01:17.892 INFO:tasks.workunit:Pulling workunits from ref 5cd0ad0001da189274219f5e829752717eb856d2 2021-05-12T09:01:17.893 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-05-12T09:01:17.893 INFO:tasks.workunit:timeout=3h 2021-05-12T09:01:17.893 INFO:tasks.workunit:cleanup=True 2021-05-12T09:01:17.894 DEBUG:teuthology.orchestra.run.smithi028:> stat -- /home/ubuntu/cephtest/mnt.0 2021-05-12T09:01:17.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-05-12T09:01:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2021-05-12T09:01:17.941 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -- /home/ubuntu/cephtest/mnt.0 2021-05-12T09:01:17.997 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2021-05-12T09:01:17.998 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2021-05-12T09:01:18.064 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5cd0ad0001da189274219f5e829752717eb856d2 2021-05-12T09:01:18.127 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-05-12T09:01:57.881 INFO:tasks.workunit.client.0.smithi028.stderr:Note: switching to '5cd0ad0001da189274219f5e829752717eb856d2'. 2021-05-12T09:01:57.881 INFO:tasks.workunit.client.0.smithi028.stderr: 2021-05-12T09:01:57.881 INFO:tasks.workunit.client.0.smithi028.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2021-05-12T09:01:57.882 INFO:tasks.workunit.client.0.smithi028.stderr:changes and commit them, and you can discard any commits you make in this 2021-05-12T09:01:57.882 INFO:tasks.workunit.client.0.smithi028.stderr:state without impacting any branches by switching back to a branch. 2021-05-12T09:01:57.882 INFO:tasks.workunit.client.0.smithi028.stderr: 2021-05-12T09:01:57.882 INFO:tasks.workunit.client.0.smithi028.stderr:If you want to create a new branch to retain commits you create, you may 2021-05-12T09:01:57.883 INFO:tasks.workunit.client.0.smithi028.stderr:do so (now or later) by using -c with the switch command. Example: 2021-05-12T09:01:57.883 INFO:tasks.workunit.client.0.smithi028.stderr: 2021-05-12T09:01:57.883 INFO:tasks.workunit.client.0.smithi028.stderr: git switch -c 2021-05-12T09:01:57.883 INFO:tasks.workunit.client.0.smithi028.stderr: 2021-05-12T09:01:57.883 INFO:tasks.workunit.client.0.smithi028.stderr:Or undo this operation with: 2021-05-12T09:01:57.884 INFO:tasks.workunit.client.0.smithi028.stderr: 2021-05-12T09:01:57.884 INFO:tasks.workunit.client.0.smithi028.stderr: git switch - 2021-05-12T09:01:57.884 INFO:tasks.workunit.client.0.smithi028.stderr: 2021-05-12T09:01:57.884 INFO:tasks.workunit.client.0.smithi028.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2021-05-12T09:01:57.885 INFO:tasks.workunit.client.0.smithi028.stderr: 2021-05-12T09:01:57.885 INFO:tasks.workunit.client.0.smithi028.stderr:HEAD is now at 5cd0ad0001d Merge branch 'negative_progress' of https://github.com/dvanders/ceph into wip-kefu-testing-2021-05-05-1449 2021-05-12T09:01:57.886 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2021-05-12T09:01:57.942 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2021-05-12T09:01:57.942 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2021-05-12T09:01:57.997 INFO:tasks.workunit:Running workunits matching osd on client.0... 2021-05-12T09:01:57.998 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2021-05-12T09:01:57.998 DEBUG:teuthology.orchestra.run.smithi028:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2021-05-12T09:01:58.065 INFO:tasks.workunit.client.0.smithi028.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T09:01:58.096 INFO:tasks.workunit.client.0.smithi028.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T09:01:58.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:01:58.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:01:58.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T09:01:58.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T09:01:58.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T09:01:58.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T09:01:58.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T09:01:58.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T09:01:58.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/bad-inc-map 2021-05-12T09:01:58.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2021-05-12T09:01:58.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2021-05-12T09:01:58.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:50637 2021-05-12T09:01:58.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:50637 2021-05-12T09:01:58.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2021-05-12T09:01:58.100 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2021-05-12T09:01:58.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=a82bc133-4872-4efa-8a3f-dadbfbbdb406 --auth-supported=none ' 2021-05-12T09:01:58.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:50637 ' 2021-05-12T09:01:58.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2021-05-12T09:01:58.105 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2021-05-12T09:01:58.105 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T09:01:58.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2021-05-12T09:01:58.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2021-05-12T09:01:58.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2021-05-12T09:01:58.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2021-05-12T09:01:58.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2021-05-12T09:01:58.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-05-12T09:01:58.107 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:01:58.107 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-05-12T09:01:58.107 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:01:58.107 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:01:58.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:01:58.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:01:58.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:01:58.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:01:58.108 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:01:58.111 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:01:58.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:01:58.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:01:58.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:01:58.112 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:01:58.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:01:58.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:01:58.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:01:58.113 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:01:58.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:01:58.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:01:58.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-05-12T09:01:58.114 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:01:58.114 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:01:58.114 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24887 2021-05-12T09:01:58.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.24887 2021-05-12T09:01:58.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:01:58.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:01:58.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2021-05-12T09:01:58.115 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:01:58.115 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:01:58.116 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24887 2021-05-12T09:01:58.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.24887 2021-05-12T09:01:58.118 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:01:58.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T09:01:58.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T09:01:58.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:01:58.119 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2021-05-12T09:01:58.119 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2021-05-12T09:01:58.119 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2021-05-12T09:01:58.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2021-05-12T09:01:58.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2021-05-12T09:01:58.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:01:58.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:01:58.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:01:58.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2021-05-12T09:01:58.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2021-05-12T09:01:58.504 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:01:58.505 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:01:58.505 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:01:58.505 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:01:58.506 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:01:58.506 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24887 2021-05-12T09:01:58.506 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.24887/$cluster-$name.asok' 2021-05-12T09:01:58.506 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.24887/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:01:58.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:01:58.749 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:01:58.750 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:01:58.750 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:01:58.750 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:01:58.750 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:01:58.751 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:01:58.751 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:01:58.751 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:01:58.751 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:01:58.752 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:01:58.752 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24887 2021-05-12T09:01:58.752 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.24887/ceph-mon.a.asok 2021-05-12T09:01:58.752 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:01:58.753 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.24887/ceph-mon.a.asok config get fsid 2021-05-12T09:01:58.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:01:58.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:01:58.855 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:01:58.855 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:01:58.855 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:01:58.855 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:01:58.856 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:01:58.856 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:01:58.856 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:01:58.856 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:01:58.856 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24887 2021-05-12T09:01:58.857 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.24887/ceph-mon.a.asok 2021-05-12T09:01:58.857 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:01:58.857 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.24887/ceph-mon.a.asok config get mon_host 2021-05-12T09:01:58.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2021-05-12T09:01:58.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2021-05-12T09:01:58.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:01:58.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:01:58.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:01:58.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2021-05-12T09:01:58.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:01:59.829 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:01:59.830 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:01:59.830 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:01:59.830 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:01:59.831 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:01:59.831 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24887 2021-05-12T09:01:59.831 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.24887/$cluster-$name.asok' 2021-05-12T09:01:59.831 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:01:59.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.24887/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:01:59.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2021-05-12T09:01:59.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2021-05-12T09:01:59.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:01:59.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:01:59.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:01:59.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2021-05-12T09:01:59.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a82bc133-4872-4efa-8a3f-dadbfbbdb406 --auth-supported=none --mon-host=127.0.0.1:50637 ' 2021-05-12T09:01:59.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:01:59.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:01:59.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:01:59.863 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2021-05-12T09:01:59.863 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2021-05-12T09:01:59.863 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:01:59.863 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:01:59.864 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-05-12T09:01:59.864 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:01:59.864 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:01:59.864 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:01:59.865 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:01:59.865 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:01:59.865 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24887 2021-05-12T09:01:59.865 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.24887/$cluster-$name.asok' 2021-05-12T09:01:59.865 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.24887/$cluster-$name.asok' 2021-05-12T09:01:59.866 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:01:59.866 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:01:59.866 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:01:59.866 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-05-12T09:01:59.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-05-12T09:01:59.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:01:59.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:01:59.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:01:59.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:01:59.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:01:59.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/bad-inc-map/0 2021-05-12T09:01:59.868 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:01:59.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=38a046c1-73e5-43d4-b05c-79d99d320d0c 2021-05-12T09:01:59.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 38a046c1-73e5-43d4-b05c-79d99d320d0c' 2021-05-12T09:01:59.869 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 38a046c1-73e5-43d4-b05c-79d99d320d0c 2021-05-12T09:01:59.870 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:01:59.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCHmZtgwD/GMxAAfReKx7dDKrNDhmG+FXCc4Q== 2021-05-12T09:01:59.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCHmZtgwD/GMxAAfReKx7dDKrNDhmG+FXCc4Q=="}' 2021-05-12T09:01:59.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 38a046c1-73e5-43d4-b05c-79d99d320d0c -i td/bad-inc-map/0/new.json 2021-05-12T09:02:00.174 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:02:00.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/bad-inc-map/0/new.json 2021-05-12T09:02:00.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=a82bc133-4872-4efa-8a3f-dadbfbbdb406 --auth-supported=none --mon-host=127.0.0.1:50637 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.24887/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCHmZtgwD/GMxAAfReKx7dDKrNDhmG+FXCc4Q== --osd-uuid 38a046c1-73e5-43d4-b05c-79d99d320d0c 2021-05-12T09:02:00.207 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:00.206+0000 7fe6850940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:00.209 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:00.208+0000 7fe6850940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:00.210 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:00.209+0000 7fe6850940c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:00.211 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:00.210+0000 7fe6850940c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-05-12T09:02:00.211 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:00.210+0000 7fe6850940c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-05-12T09:02:00.212 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:00.210+0000 7fe6850940c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-05-12T09:02:00.247 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:00.246+0000 7fe6850940c0 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2021-05-12T09:02:04.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/bad-inc-map/0/keyring 2021-05-12T09:02:04.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:02:04.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:02:04.821 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:02:04.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:02:05.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:02:05.052 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:02:05.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=a82bc133-4872-4efa-8a3f-dadbfbbdb406 --auth-supported=none --mon-host=127.0.0.1:50637 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.24887/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:02:05.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:02:05.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:02:05.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:02:05.071 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:05.071+0000 7f08885ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:05.074 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:05.073+0000 7f08885ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:05.074 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:05.074+0000 7f08885ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:05.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:02:05.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:02:05.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:02:05.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:02:05.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:02:05.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:05.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:02:05.276 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:02:05.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:05.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:02:05.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:02:05.591 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:05.590+0000 7f08885ff0c0 -1 Falling back to public interface 2021-05-12T09:02:06.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:02:06.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:06.511 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:02:06.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:02:06.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:06.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:02:06.894 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:02:07.809 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:07.808+0000 7f08885ff0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:02:07.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:02:07.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:07.896 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:02:07.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:02:07.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:07.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:02:08.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:02:09.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:02:09.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:09.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:02:09.283 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:02:09.284 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:09.284 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:02:09.666 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/25954,v1:127.0.0.1:6803/25954] [v2:127.0.0.1:6804/25954,v1:127.0.0.1:6805/25954] exists,up 38a046c1-73e5-43d4-b05c-79d99d320d0c 2021-05-12T09:02:09.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:02:09.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:02:09.671 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:02:09.671 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2021-05-12T09:02:09.671 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2021-05-12T09:02:09.671 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:02:09.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:02:09.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:02:09.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2021-05-12T09:02:09.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a82bc133-4872-4efa-8a3f-dadbfbbdb406 --auth-supported=none --mon-host=127.0.0.1:50637 ' 2021-05-12T09:02:09.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:02:09.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:02:09.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:02:09.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2021-05-12T09:02:09.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2021-05-12T09:02:09.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:02:09.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:02:09.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-05-12T09:02:09.674 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:02:09.674 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:02:09.675 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:02:09.675 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:02:09.675 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:02:09.676 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24887 2021-05-12T09:02:09.676 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.24887/$cluster-$name.asok' 2021-05-12T09:02:09.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.24887/$cluster-$name.asok' 2021-05-12T09:02:09.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:02:09.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:02:09.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:02:09.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-05-12T09:02:09.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-05-12T09:02:09.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:02:09.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:02:09.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:02:09.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:02:09.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:02:09.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/bad-inc-map/1 2021-05-12T09:02:09.679 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:02:09.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=e4bf56d9-281d-4027-aca2-ff958f835f55 2021-05-12T09:02:09.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 e4bf56d9-281d-4027-aca2-ff958f835f55' 2021-05-12T09:02:09.680 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 e4bf56d9-281d-4027-aca2-ff958f835f55 2021-05-12T09:02:09.681 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:02:09.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCRmZtgzcGoKBAAkB33rDGL7XzGfAC4FsHkIw== 2021-05-12T09:02:09.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCRmZtgzcGoKBAAkB33rDGL7XzGfAC4FsHkIw=="}' 2021-05-12T09:02:09.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new e4bf56d9-281d-4027-aca2-ff958f835f55 -i td/bad-inc-map/1/new.json 2021-05-12T09:02:10.150 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:02:10.161 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/bad-inc-map/1/new.json 2021-05-12T09:02:10.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=a82bc133-4872-4efa-8a3f-dadbfbbdb406 --auth-supported=none --mon-host=127.0.0.1:50637 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.24887/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCRmZtgzcGoKBAAkB33rDGL7XzGfAC4FsHkIw== --osd-uuid e4bf56d9-281d-4027-aca2-ff958f835f55 2021-05-12T09:02:10.183 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:10.183+0000 7f8ffc3750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:10.185 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:10.185+0000 7f8ffc3750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:10.186 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:10.186+0000 7f8ffc3750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:10.187 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:10.187+0000 7f8ffc3750c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-05-12T09:02:10.187 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:10.187+0000 7f8ffc3750c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-05-12T09:02:10.187 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:10.187+0000 7f8ffc3750c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-05-12T09:02:10.229 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:10.229+0000 7f8ffc3750c0 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2021-05-12T09:02:14.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/bad-inc-map/1/keyring 2021-05-12T09:02:14.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:02:14.646 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:02:14.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:02:14.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:02:15.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:02:15.155 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:02:15.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=a82bc133-4872-4efa-8a3f-dadbfbbdb406 --auth-supported=none --mon-host=127.0.0.1:50637 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.24887/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:02:15.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:02:15.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:02:15.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:02:15.174 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:15.175+0000 7f4165be50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:15.177 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:15.177+0000 7f4165be50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:15.178 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:15.178+0000 7f4165be50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:15.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:02:15.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:02:15.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:02:15.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:02:15.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:02:15.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:15.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:02:15.548 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:02:15.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:15.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:02:15.694 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:15.695+0000 7f4165be50c0 -1 Falling back to public interface 2021-05-12T09:02:15.935 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:02:16.937 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:02:16.937 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:16.937 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:02:16.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:02:16.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:16.939 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:02:17.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:02:17.854 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:17.854+0000 7f4165be50c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:02:18.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:02:18.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:18.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:02:18.328 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:02:18.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:18.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:02:18.720 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/27429,v1:127.0.0.1:6811/27429] [v2:127.0.0.1:6812/27429,v1:127.0.0.1:6813/27429] exists,up e4bf56d9-281d-4027-aca2-ff958f835f55 2021-05-12T09:02:18.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:02:18.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:02:18.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:02:18.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2021-05-12T09:02:18.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2021-05-12T09:02:18.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:02:18.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T09:02:18.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:02:18.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2021-05-12T09:02:18.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a82bc133-4872-4efa-8a3f-dadbfbbdb406 --auth-supported=none --mon-host=127.0.0.1:50637 ' 2021-05-12T09:02:18.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:02:18.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:02:18.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:02:18.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2021-05-12T09:02:18.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2021-05-12T09:02:18.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:02:18.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:02:18.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-05-12T09:02:18.729 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:02:18.729 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:02:18.729 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:02:18.730 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:02:18.730 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:02:18.730 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24887 2021-05-12T09:02:18.730 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.24887/$cluster-$name.asok' 2021-05-12T09:02:18.731 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.24887/$cluster-$name.asok' 2021-05-12T09:02:18.731 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:02:18.731 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:02:18.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:02:18.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-05-12T09:02:18.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-05-12T09:02:18.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:02:18.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:02:18.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:02:18.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:02:18.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:02:18.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/bad-inc-map/2 2021-05-12T09:02:18.734 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:02:18.734 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=2a615a9f-6590-4fd2-ac2c-222282ca1c35 2021-05-12T09:02:18.734 INFO:tasks.workunit.client.0.smithi028.stdout:add osd2 2a615a9f-6590-4fd2-ac2c-222282ca1c35 2021-05-12T09:02:18.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 2a615a9f-6590-4fd2-ac2c-222282ca1c35' 2021-05-12T09:02:18.735 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:02:18.738 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCamZtgymT+KxAALouic8d9UCKmLZgoV2+pFQ== 2021-05-12T09:02:18.738 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCamZtgymT+KxAALouic8d9UCKmLZgoV2+pFQ=="}' 2021-05-12T09:02:18.738 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 2a615a9f-6590-4fd2-ac2c-222282ca1c35 -i td/bad-inc-map/2/new.json 2021-05-12T09:02:19.213 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:02:19.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/bad-inc-map/2/new.json 2021-05-12T09:02:19.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=a82bc133-4872-4efa-8a3f-dadbfbbdb406 --auth-supported=none --mon-host=127.0.0.1:50637 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.24887/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCamZtgymT+KxAALouic8d9UCKmLZgoV2+pFQ== --osd-uuid 2a615a9f-6590-4fd2-ac2c-222282ca1c35 2021-05-12T09:02:19.247 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:19.247+0000 7fc86a93b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:19.249 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:19.249+0000 7fc86a93b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:19.250 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:19.250+0000 7fc86a93b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:19.251 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:19.251+0000 7fc86a93b0c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-05-12T09:02:19.251 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:19.251+0000 7fc86a93b0c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-05-12T09:02:19.251 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:19.251+0000 7fc86a93b0c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-05-12T09:02:19.317 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:19.318+0000 7fc86a93b0c0 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2021-05-12T09:02:23.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/bad-inc-map/2/keyring 2021-05-12T09:02:23.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:02:23.836 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd2 key to auth repository 2021-05-12T09:02:23.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T09:02:23.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:02:24.347 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:02:24.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T09:02:24.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=a82bc133-4872-4efa-8a3f-dadbfbbdb406 --auth-supported=none --mon-host=127.0.0.1:50637 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.24887/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:02:24.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:02:24.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:02:24.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:02:24.367 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:24.367+0000 7fb6ad64e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:24.369 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:24.369+0000 7fb6ad64e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:24.370 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:24.370+0000 7fb6ad64e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:24.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T09:02:24.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:02:24.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:02:24.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:02:24.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:02:24.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:24.727 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:02:24.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:02:24.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:24.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:02:24.887 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:24.887+0000 7fb6ad64e0c0 -1 Falling back to public interface 2021-05-12T09:02:25.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:02:26.122 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:02:26.122 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:26.123 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:02:26.123 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:02:26.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:26.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:02:26.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:02:27.288 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:27.288+0000 7fb6ad64e0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T09:02:27.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:02:27.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:27.511 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:02:27.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:02:27.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:27.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:02:28.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:02:29.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:02:29.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:29.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:02:29.343 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:02:29.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:29.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:02:29.887 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:02:30.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:02:30.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:30.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T09:02:30.889 INFO:tasks.workunit.client.0.smithi028.stdout:4 2021-05-12T09:02:30.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:30.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:02:31.273 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/28814,v1:127.0.0.1:6819/28814] [v2:127.0.0.1:6820/28814,v1:127.0.0.1:6821/28814] exists,up 2a615a9f-6590-4fd2-ac2c-222282ca1c35 2021-05-12T09:02:31.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:02:31.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:02:31.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:02:31.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2021-05-12T09:02:31.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2021-05-12T09:02:31.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2021-05-12T09:02:32.489 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'foo' already exists 2021-05-12T09:02:32.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:02:33.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2021-05-12T09:02:34.665 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 2 min_size to 1 2021-05-12T09:02:34.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2021-05-12T09:02:35.806 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 2 min_size to 2 2021-05-12T09:02:35.816 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2021-05-12T09:02:40.817 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2021-05-12T09:02:40.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2021-05-12T09:02:40.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:02:40.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:02:40.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:02:40.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:02:40.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 10 )) 2021-05-12T09:02:40.820 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:02:40.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:02:40.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:40.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:02:41.202 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 28 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/25954,v1:127.0.0.1:6803/25954] [v2:127.0.0.1:6804/25954,v1:127.0.0.1:6805/25954] exists,up 38a046c1-73e5-43d4-b05c-79d99d320d0c 2021-05-12T09:02:41.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:02:41.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:02:41.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:02:41.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2021-05-12T09:02:41.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2021-05-12T09:02:41.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:02:41.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:02:41.206 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:02:41.206 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:02:41.206 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 10 )) 2021-05-12T09:02:41.206 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:02:41.207 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:02:41.208 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:41.208 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:02:41.607 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 28 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/27429,v1:127.0.0.1:6811/27429] [v2:127.0.0.1:6812/27429,v1:127.0.0.1:6813/27429] exists,up e4bf56d9-281d-4027-aca2-ff958f835f55 2021-05-12T09:02:41.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:02:41.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:02:41.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:02:41.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2021-05-12T09:02:41.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2021-05-12T09:02:41.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:02:41.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:02:41.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:02:41.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:02:41.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 10 )) 2021-05-12T09:02:41.612 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:02:41.613 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:02:41.614 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:41.614 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:02:41.991 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 28 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/28814,v1:127.0.0.1:6819/28814] [v2:127.0.0.1:6820/28814,v1:127.0.0.1:6821/28814] exists,up 2a615a9f-6590-4fd2-ac2c-222282ca1c35 2021-05-12T09:02:41.992 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:02:41.992 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:02:41.992 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:02:41.992 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2021-05-12T09:02:41.993 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:31.819+0000 7fb694559700 -1 osd.2 17 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.993 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:32.498+0000 7fb692555700 -1 osd.2 19 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.994 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:32.499+0000 7fb692555700 -1 osd.2 19 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.994 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:32.920+0000 7fb692555700 -1 osd.2 20 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.994 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:32.985+0000 7fb694559700 -1 osd.2 20 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.994 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:33.688+0000 7fb692555700 -1 osd.2 22 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.995 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:34.149+0000 7fb692555700 -1 osd.2 23 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.995 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:34.306+0000 7fb694559700 -1 osd.2 24 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.995 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:34.660+0000 7fb694559700 -1 osd.2 25 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.995 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:34.661+0000 7fb694559700 -1 osd.2 25 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.996 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:35.355+0000 7fb692555700 -1 osd.2 26 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.996 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:35.591+0000 7fb692555700 -1 osd.2 27 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.996 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:35.800+0000 7fb694559700 -1 osd.2 28 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.996 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:36.001+0000 7fb694559700 -1 osd.2 29 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.997 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:36.003+0000 7fb694559700 -1 osd.2 29 handle_osd_map injecting map crc failure 2021-05-12T09:02:41.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2021-05-12T09:02:41.998 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:31.820+0000 7fb694559700 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2021-05-12T09:02:41.998 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:32.499+0000 7fb692555700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2021-05-12T09:02:41.998 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:32.499+0000 7fb692555700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2021-05-12T09:02:41.999 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:32.921+0000 7fb692555700 10 osd.2 20 handle_osd_map bailing because last < start (20<21) 2021-05-12T09:02:41.999 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:32.986+0000 7fb694559700 10 osd.2 20 handle_osd_map bailing because last < start (21<22) 2021-05-12T09:02:41.999 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:33.688+0000 7fb692555700 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2021-05-12T09:02:41.999 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:34.149+0000 7fb692555700 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2021-05-12T09:02:42.000 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:34.306+0000 7fb694559700 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2021-05-12T09:02:42.000 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:34.660+0000 7fb694559700 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2021-05-12T09:02:42.000 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:34.662+0000 7fb694559700 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2021-05-12T09:02:42.000 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:35.355+0000 7fb692555700 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2021-05-12T09:02:42.001 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:35.592+0000 7fb692555700 10 osd.2 27 handle_osd_map bailing because last < start (27<28) 2021-05-12T09:02:42.001 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:35.800+0000 7fb694559700 10 osd.2 28 handle_osd_map bailing because last < start (28<29) 2021-05-12T09:02:42.001 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:36.002+0000 7fb694559700 10 osd.2 29 handle_osd_map bailing because last < start (29<30) 2021-05-12T09:02:42.001 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:02:36.003+0000 7fb694559700 10 osd.2 29 handle_osd_map bailing because last < start (29<30) 2021-05-12T09:02:42.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2021-05-12T09:02:42.002 INFO:tasks.workunit.client.0.smithi028.stdout:success 2021-05-12T09:02:42.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2021-05-12T09:02:42.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2021-05-12T09:02:42.004 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2021-05-12T09:02:42.496 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'foo' does not exist 2021-05-12T09:02:42.505 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2021-05-12T09:02:42.506 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:02:42.506 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:02:42.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:02:42.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:02:42.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:02:42.615 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:02:42.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2021-05-12T09:02:42.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-05-12T09:02:42.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:02:42.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-05-12T09:02:42.616 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:02:42.617 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:02:42.617 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:02:42.617 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:02:42.617 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:02:42.619 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:02:42.620 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:02:42.620 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:02:42.621 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:02:42.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:02:42.622 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:02:42.622 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:02:42.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:02:42.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:02:42.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:02:42.624 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:02:42.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:02:42.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:02:42.626 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-05-12T09:02:42.632 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:02:42.632 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:02:42.632 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24887 2021-05-12T09:02:42.632 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.24887 2021-05-12T09:02:42.634 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:02:42.634 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:02:42.634 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T09:02:42.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/bad-inc-map 0 2021-05-12T09:02:42.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-05-12T09:02:42.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T09:02:42.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-05-12T09:02:42.636 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:02:42.636 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:02:42.636 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:02:42.636 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:02:42.636 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:02:42.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:02:42.638 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:02:42.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:02:42.638 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:02:42.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:02:42.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:02:42.639 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:02:42.640 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:02:42.640 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:02:42.640 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:02:42.641 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:02:42.641 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:02:42.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T09:02:42.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-05-12T09:02:42.643 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:02:42.643 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:02:42.644 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.24887 2021-05-12T09:02:42.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.24887 2021-05-12T09:02:42.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:02:42.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:02:42.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T09:02:42.646 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T09:02:42.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T09:02:42.712 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2021-05-12T09:02:42.712 DEBUG:teuthology.orchestra.run.smithi028:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2021-05-12T09:02:42.773 INFO:tasks.workunit.client.0.smithi028.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T09:02:42.782 INFO:tasks.workunit.client.0.smithi028.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T09:02:42.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:02:42.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:02:42.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T09:02:42.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T09:02:42.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T09:02:42.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T09:02:42.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T09:02:42.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T09:02:42.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/divergent-priors 2021-05-12T09:02:42.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2021-05-12T09:02:42.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2021-05-12T09:02:42.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2021-05-12T09:02:42.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2021-05-12T09:02:42.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2021-05-12T09:02:42.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2021-05-12T09:02:42.786 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2021-05-12T09:02:42.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2021-05-12T09:02:42.787 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2021-05-12T09:02:42.787 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2021-05-12T09:02:42.794 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2021-05-12T09:02:42.794 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2021-05-12T09:02:42.795 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2021-05-12T09:02:42.795 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2021-05-12T09:02:42.795 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2021-05-12T09:02:42.795 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2021-05-12T09:02:42.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2021-05-12T09:02:42.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2021-05-12T09:02:42.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2021-05-12T09:02:42.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2021-05-12T09:02:42.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2021-05-12T09:02:42.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2021-05-12T09:02:42.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2021-05-12T09:02:42.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2021-05-12T09:02:42.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2021-05-12T09:02:42.798 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2021-05-12T09:02:42.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none ' 2021-05-12T09:02:42.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2021-05-12T09:02:42.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2021-05-12T09:02:42.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2021-05-12T09:02:42.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:02:42.799 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2021-05-12T09:02:42.800 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T09:02:42.800 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2021-05-12T09:02:42.800 INFO:tasks.workunit.client.0.smithi028.stderr:TEST_divergent_2 2021-05-12T09:02:42.800 INFO:tasks.workunit.client.0.smithi028.stderr:TEST_divergent_3 2021-05-12T09:02:42.801 INFO:tasks.workunit.client.0.smithi028.stderr:TEST_divergent_ec' 2021-05-12T09:02:42.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-05-12T09:02:42.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-05-12T09:02:42.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-05-12T09:02:42.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-05-12T09:02:42.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T09:02:42.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:02:42.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T09:02:42.802 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:02:42.803 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:02:42.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:02:42.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:02:42.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:02:42.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:02:42.804 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:02:42.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:02:42.804 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:02:42.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:02:42.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:02:42.805 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:02:42.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:02:42.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:02:42.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:02:42.806 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:02:42.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:02:42.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:02:42.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T09:02:42.809 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:02:42.810 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:02:42.810 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:02:42.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29719 2021-05-12T09:02:42.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:02:42.811 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:02:42.811 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-05-12T09:02:42.811 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:02:42.811 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:02:42.811 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:02:42.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.29719 2021-05-12T09:02:42.812 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:02:42.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T09:02:42.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T09:02:42.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:02:42.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-05-12T09:02:42.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2021-05-12T09:02:42.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2021-05-12T09:02:42.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2021-05-12T09:02:42.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2021-05-12T09:02:42.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2021-05-12T09:02:42.814 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2021-05-12T09:02:42.814 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2021-05-12T09:02:42.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2021-05-12T09:02:42.821 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:02:42.821 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:02:42.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2021-05-12T09:02:42.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-05-12T09:02:42.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:02:42.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:02:42.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:02:42.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-05-12T09:02:42.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-05-12T09:02:43.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:02:43.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:02:43.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:02:43.144 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:02:43.144 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:02:43.144 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:02:43.144 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:02:43.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:02:43.403 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:02:43.405 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:02:43.405 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:02:43.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:02:43.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:02:43.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:02:43.406 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:02:43.407 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:02:43.407 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:02:43.407 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:02:43.407 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:02:43.407 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:02:43.408 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-mon.a.asok 2021-05-12T09:02:43.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:02:43.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29719/ceph-mon.a.asok config get fsid 2021-05-12T09:02:43.507 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:02:43.507 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:02:43.508 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:02:43.508 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:02:43.508 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:02:43.508 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:02:43.509 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:02:43.509 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:02:43.509 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:02:43.509 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:02:43.510 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:02:43.510 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-mon.a.asok 2021-05-12T09:02:43.510 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:02:43.510 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29719/ceph-mon.a.asok config get mon_host 2021-05-12T09:02:43.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2021-05-12T09:02:43.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-05-12T09:02:43.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:02:43.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:02:43.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:02:43.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-05-12T09:02:43.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:02:44.074 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:02:44.075 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:02:44.075 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:02:44.075 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:02:44.075 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:02:44.076 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:02:44.076 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:02:44.076 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:02:44.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:02:44.107 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-05-12T09:02:44.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2021-05-12T09:02:44.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T09:02:44.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:02:44.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:02:44.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:02:44.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-05-12T09:02:44.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:02:44.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:02:44.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:02:44.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:02:44.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T09:02:44.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T09:02:44.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:02:44.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:02:44.111 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:02:44.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:02:44.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:02:44.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:02:44.111 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:02:44.112 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:02:44.112 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:02:44.112 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:02:44.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:02:44.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:02:44.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:02:44.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:02:44.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:02:44.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:02:44.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:02:44.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:02:44.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:02:44.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:02:44.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:02:44.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/0 2021-05-12T09:02:44.115 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:02:44.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=76b38569-e8bc-43f9-a984-1867943d6e8d 2021-05-12T09:02:44.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 76b38569-e8bc-43f9-a984-1867943d6e8d' 2021-05-12T09:02:44.116 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 76b38569-e8bc-43f9-a984-1867943d6e8d 2021-05-12T09:02:44.116 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:02:44.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQC0mZtg7w6SBxAAPZ4CJQYxWcjrWOE5ubXhIg== 2021-05-12T09:02:44.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQC0mZtg7w6SBxAAPZ4CJQYxWcjrWOE5ubXhIg=="}' 2021-05-12T09:02:44.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 76b38569-e8bc-43f9-a984-1867943d6e8d -i td/divergent-priors/0/new.json 2021-05-12T09:02:44.428 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:02:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/0/new.json 2021-05-12T09:02:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC0mZtg7w6SBxAAPZ4CJQYxWcjrWOE5ubXhIg== --osd-uuid 76b38569-e8bc-43f9-a984-1867943d6e8d 2021-05-12T09:02:44.472 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:44.471+0000 7f1efde6c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:44.473+0000 7f1efde6c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:44.474+0000 7f1efde6c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:44.476+0000 7f1efde6c0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T09:02:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:44.476+0000 7f1efde6c0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T09:02:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:44.476+0000 7f1efde6c0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T09:02:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:44.517+0000 7f1efde6c0c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-05-12T09:02:48.852 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-05-12T09:02:48.853 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:02:48.853 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:02:48.854 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:02:48.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:02:49.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:02:49.126 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:02:49.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:02:49.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:02:49.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:02:49.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:02:49.155 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:49.155+0000 7f467ebad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:49.157 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:49.157+0000 7f467ebad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:49.159 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:49.159+0000 7f467ebad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:49.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:02:49.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:02:49.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:02:49.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:02:49.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:02:49.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:49.527 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:02:49.527 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:02:49.528 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:49.528 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:02:49.678 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:49.678+0000 7f467ebad0c0 -1 Falling back to public interface 2021-05-12T09:02:49.908 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:02:50.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:02:50.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:50.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:02:50.910 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:02:50.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:50.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:02:51.310 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:02:51.879 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:51.879+0000 7f467ebad0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:02:52.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:02:52.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:52.311 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:02:52.312 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:02:52.312 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:52.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:02:52.731 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/30846,v1:127.0.0.1:6803/30846] [v2:127.0.0.1:6804/30846,v1:127.0.0.1:6805/30846] exists,up 76b38569-e8bc-43f9-a984-1867943d6e8d 2021-05-12T09:02:52.737 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:02:52.737 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:02:52.737 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:02:52.737 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-05-12T09:02:52.738 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2021-05-12T09:02:52.738 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T09:02:52.738 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:02:52.738 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:02:52.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:02:52.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-05-12T09:02:52.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:02:52.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:02:52.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:02:52.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:02:52.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T09:02:52.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T09:02:52.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:02:52.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:02:52.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:02:52.741 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:02:52.741 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:02:52.741 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:02:52.742 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:02:52.742 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:02:52.743 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:02:52.743 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:02:52.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:02:52.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:02:52.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:02:52.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:02:52.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:02:52.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:02:52.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:02:52.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:02:52.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:02:52.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:02:52.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:02:52.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/1 2021-05-12T09:02:52.746 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:02:52.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=fcbcbfab-6930-4442-a86e-6c4dd2cd2b5c 2021-05-12T09:02:52.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 fcbcbfab-6930-4442-a86e-6c4dd2cd2b5c' 2021-05-12T09:02:52.747 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 fcbcbfab-6930-4442-a86e-6c4dd2cd2b5c 2021-05-12T09:02:52.747 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:02:52.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQC8mZtgWnz1LBAAvCNr7K8287+JnLdPzX/e3w== 2021-05-12T09:02:52.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQC8mZtgWnz1LBAAvCNr7K8287+JnLdPzX/e3w=="}' 2021-05-12T09:02:52.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new fcbcbfab-6930-4442-a86e-6c4dd2cd2b5c -i td/divergent-priors/1/new.json 2021-05-12T09:02:53.244 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:02:53.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/1/new.json 2021-05-12T09:02:53.257 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC8mZtgWnz1LBAAvCNr7K8287+JnLdPzX/e3w== --osd-uuid fcbcbfab-6930-4442-a86e-6c4dd2cd2b5c 2021-05-12T09:02:53.288 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:53.288+0000 7f9a48ab90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:53.290 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:53.290+0000 7f9a48ab90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:53.291 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:53.291+0000 7f9a48ab90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:53.292 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:53.292+0000 7f9a48ab90c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T09:02:53.292 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:53.292+0000 7f9a48ab90c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T09:02:53.292 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:53.292+0000 7f9a48ab90c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T09:02:53.367 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:53.367+0000 7f9a48ab90c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-05-12T09:02:58.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-05-12T09:02:58.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:02:58.200 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:02:58.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:02:58.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:02:58.724 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:02:58.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:02:58.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:02:58.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:02:58.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:02:58.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:02:58.754 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:58.754+0000 7fa2ce3fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:58.755 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:58.756+0000 7fa2ce3fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:58.757 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:58.757+0000 7fa2ce3fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:02:59.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:02:59.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:02:59.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:02:59.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:02:59.130 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:02:59.130 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:02:59.130 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:02:59.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:02:59.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:02:59.132 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:02:59.277 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:02:59.277+0000 7fa2ce3fd0c0 -1 Falling back to public interface 2021-05-12T09:02:59.519 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:03:00.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:03:00.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:03:00.522 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:03:00.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:03:00.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:03:00.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:03:00.920 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:03:01.536 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:01.537+0000 7fa2ce3fd0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:03:01.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:03:01.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:03:01.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:03:01.923 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:03:01.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:03:01.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:03:02.317 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/32231,v1:127.0.0.1:6811/32231] [v2:127.0.0.1:6812/32231,v1:127.0.0.1:6813/32231] exists,up fcbcbfab-6930-4442-a86e-6c4dd2cd2b5c 2021-05-12T09:03:02.322 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:03:02.322 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:03:02.322 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:03:02.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-05-12T09:03:02.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2021-05-12T09:03:02.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T09:03:02.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:03:02.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T09:03:02.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:03:02.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-05-12T09:03:02.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:03:02.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:03:02.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:03:02.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:03:02.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T09:03:02.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T09:03:02.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:03:02.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:03:02.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:03:02.326 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:03:02.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:03:02.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:03:02.327 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:03:02.327 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:03:02.328 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:03:02.328 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:03:02.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:03:02.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:03:02.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:03:02.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:03:02.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:03:02.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:03:02.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:03:02.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:03:02.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:03:02.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:03:02.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:03:02.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/2 2021-05-12T09:03:02.331 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:03:02.331 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=786b0ee9-ebdd-46c7-9baf-4db98735ea54 2021-05-12T09:03:02.331 INFO:tasks.workunit.client.0.smithi028.stdout:add osd2 786b0ee9-ebdd-46c7-9baf-4db98735ea54 2021-05-12T09:03:02.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 786b0ee9-ebdd-46c7-9baf-4db98735ea54' 2021-05-12T09:03:02.332 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:03:02.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDGmZtg9jprFBAAukZBiDlNyT4SpgJWmwnQUQ== 2021-05-12T09:03:02.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDGmZtg9jprFBAAukZBiDlNyT4SpgJWmwnQUQ=="}' 2021-05-12T09:03:02.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 786b0ee9-ebdd-46c7-9baf-4db98735ea54 -i td/divergent-priors/2/new.json 2021-05-12T09:03:02.868 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:03:02.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/2/new.json 2021-05-12T09:03:02.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDGmZtg9jprFBAAukZBiDlNyT4SpgJWmwnQUQ== --osd-uuid 786b0ee9-ebdd-46c7-9baf-4db98735ea54 2021-05-12T09:03:02.912 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:02.912+0000 7f9fc7fb40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:03:02.913 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:02.914+0000 7f9fc7fb40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:03:02.914 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:02.915+0000 7f9fc7fb40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:03:02.915 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:02.915+0000 7f9fc7fb40c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T09:03:02.915 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:02.915+0000 7f9fc7fb40c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T09:03:02.916 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:02.915+0000 7f9fc7fb40c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T09:03:02.990 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:02.991+0000 7f9fc7fb40c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-05-12T09:03:07.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-05-12T09:03:07.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:03:07.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T09:03:07.542 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd2 key to auth repository 2021-05-12T09:03:07.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:03:08.066 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T09:03:08.067 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:03:08.067 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:03:08.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:03:08.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:03:08.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:03:08.095 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:08.094+0000 7f14afa1b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:03:08.114 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:08.113+0000 7f14afa1b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:03:08.116 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:08.115+0000 7f14afa1b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:03:08.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T09:03:08.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:03:08.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:03:08.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:03:08.455 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:03:08.455 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:03:08.455 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:03:08.455 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:03:08.456 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:03:08.456 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:03:08.636 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:08.635+0000 7f14afa1b0c0 -1 Falling back to public interface 2021-05-12T09:03:08.858 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:03:09.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:03:09.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:03:09.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:03:09.860 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:03:09.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:03:09.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:03:10.253 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:03:10.825 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:10.825+0000 7f14afa1b0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T09:03:11.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:03:11.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:03:11.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:03:11.256 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:03:11.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:03:11.257 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:03:11.672 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/33624,v1:127.0.0.1:6819/33624] [v2:127.0.0.1:6820/33624,v1:127.0.0.1:6821/33624] exists,up 786b0ee9-ebdd-46c7-9baf-4db98735ea54 2021-05-12T09:03:11.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:03:11.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:03:11.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:03:11.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2021-05-12T09:03:12.533 INFO:tasks.workunit.client.0.smithi028.stderr:noout is set 2021-05-12T09:03:12.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2021-05-12T09:03:13.428 INFO:tasks.workunit.client.0.smithi028.stderr:noin is set 2021-05-12T09:03:13.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2021-05-12T09:03:14.241 INFO:tasks.workunit.client.0.smithi028.stderr:nodown is set 2021-05-12T09:03:14.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2021-05-12T09:03:14.251 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-05-12T09:03:14.905 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'test' already exists 2021-05-12T09:03:14.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:03:15.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2021-05-12T09:03:16.765 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 2 size to 3 2021-05-12T09:03:16.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2021-05-12T09:03:17.535 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 2 min_size to 2 2021-05-12T09:03:17.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2021-05-12T09:03:17.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:03:17.546 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:03:17.935 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:03:17.935 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:03:17.936 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:03:17.936 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:03:17.936 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:03:17.937 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:03:18.076 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836487 2021-05-12T09:03:18.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836487 2021-05-12T09:03:18.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487' 2021-05-12T09:03:18.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:03:18.078 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:03:18.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672966 2021-05-12T09:03:18.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672966 2021-05-12T09:03:18.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-42949672966' 2021-05-12T09:03:18.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:03:18.219 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:03:18.360 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509445 2021-05-12T09:03:18.361 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509445 2021-05-12T09:03:18.361 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-42949672966 2-64424509445' 2021-05-12T09:03:18.361 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:03:18.362 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-05-12T09:03:18.362 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:03:18.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:03:18.363 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836487 2021-05-12T09:03:18.363 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:03:18.364 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836487 2021-05-12T09:03:18.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-05-12T09:03:18.365 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836487 2021-05-12T09:03:18.366 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:03:18.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836485 -lt 21474836487 2021-05-12T09:03:18.756 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:03:19.756 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:03:19.757 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:03:20.159 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836487 2021-05-12T09:03:20.159 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:03:20.160 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672966 2021-05-12T09:03:20.160 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:03:20.161 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:03:20.162 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672966 2021-05-12T09:03:20.162 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:03:20.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672966 2021-05-12T09:03:20.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-05-12T09:03:20.164 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672966 2021-05-12T09:03:20.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:03:20.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672966 -lt 42949672966 2021-05-12T09:03:20.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:03:20.553 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509445 2021-05-12T09:03:20.553 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:03:20.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:03:20.555 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509445 2021-05-12T09:03:20.555 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:03:20.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509445 2021-05-12T09:03:20.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2021-05-12T09:03:20.557 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509445 2021-05-12T09:03:20.557 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:03:20.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509445 -lt 64424509445 2021-05-12T09:03:20.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2021-05-12T09:03:20.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:03:20.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:03:20.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:03:20.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:03:20.984 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T09:03:20.985 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:03:20.985 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:03:20.985 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:03:20.986 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:03:20.986 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:03:21.074 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:03:21.074 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:03:21.075 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:03:21.075 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:03:21.075 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:03:21.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:03:21.480 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:03:21.480 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:03:21.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:03:21.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:03:21.481 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:03:21.620 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836490 2021-05-12T09:03:21.620 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836490 2021-05-12T09:03:21.620 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490' 2021-05-12T09:03:21.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:03:21.621 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:03:21.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672968 2021-05-12T09:03:21.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672968 2021-05-12T09:03:21.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2021-05-12T09:03:21.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:03:21.764 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:03:21.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509447 2021-05-12T09:03:21.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509447 2021-05-12T09:03:21.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509447' 2021-05-12T09:03:21.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:03:21.905 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-05-12T09:03:21.906 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:03:21.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:03:21.907 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836490 2021-05-12T09:03:21.907 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:03:21.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836490 2021-05-12T09:03:21.909 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836490 2021-05-12T09:03:21.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-05-12T09:03:21.910 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:03:22.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836490 2021-05-12T09:03:22.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:03:23.392 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:03:23.393 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:03:23.857 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836490 -lt 21474836490 2021-05-12T09:03:23.858 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:03:23.859 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2021-05-12T09:03:23.859 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:03:23.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:03:23.861 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672968 2021-05-12T09:03:23.861 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:03:23.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672968 2021-05-12T09:03:23.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-05-12T09:03:23.863 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672968 2021-05-12T09:03:23.863 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:03:24.251 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672969 -lt 42949672968 2021-05-12T09:03:24.251 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:03:24.252 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509447 2021-05-12T09:03:24.252 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:03:24.253 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:03:24.254 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509447 2021-05-12T09:03:24.254 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:03:24.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509447 2021-05-12T09:03:24.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2021-05-12T09:03:24.256 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509447 2021-05-12T09:03:24.256 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:03:24.640 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509448 -lt 64424509447 2021-05-12T09:03:24.641 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:03:24.641 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:03:24.642 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:03:25.172 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:03:25.172 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:03:25.173 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:03:25.173 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:03:25.174 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:03:25.174 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:03:25.174 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:03:25.174 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:03:25.550 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T09:03:25.551 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:03:25.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:03:25.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:03:26.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T09:03:26.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:03:26.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:03:26.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2021-05-12T09:03:26.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2021-05-12T09:03:26.430 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:03:26.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=2 2021-05-12T09:03:26.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 2' 2021-05-12T09:03:26.440 INFO:tasks.workunit.client.0.smithi028.stdout:primary and soon to be divergent is 2 2021-05-12T09:03:26.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2021-05-12T09:03:26.803 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:03:26.803 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T09:03:15.790700+0000 0'0 30:13 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:03:14.594281+0000 0'0 2021-05-12T09:03:14.594281+0000 0 2021-05-12T09:03:26.804 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:03:13.981969+0000 31'62 31:102 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:03:13.031503+0000 0'0 2021-05-12T09:03:13.031503+0000 0 2021-05-12T09:03:26.804 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:03:26.804 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:03:26.805 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:03:26.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2021-05-12T09:03:26.816 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-05-12T09:03:26.816 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 2 ']' 2021-05-12T09:03:26.816 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2021-05-12T09:03:26.816 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-05-12T09:03:26.817 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 2 ']' 2021-05-12T09:03:26.817 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 1' 2021-05-12T09:03:26.817 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-05-12T09:03:26.817 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 2 ']' 2021-05-12T09:03:26.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2021-05-12T09:03:26.818 INFO:tasks.workunit.client.0.smithi028.stdout:writing initial objects 2021-05-12T09:03:26.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2021-05-12T09:03:26.819 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2021-05-12T09:03:26.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:26.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2021-05-12T09:03:26.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:26.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2021-05-12T09:03:27.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:27.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2021-05-12T09:03:27.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:27.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2021-05-12T09:03:27.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:27.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2021-05-12T09:03:27.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:27.403 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2021-05-12T09:03:27.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:27.504 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2021-05-12T09:03:27.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:27.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2021-05-12T09:03:27.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:27.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2021-05-12T09:03:27.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:27.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2021-05-12T09:03:27.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:27.932 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2021-05-12T09:03:28.066 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:28.067 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2021-05-12T09:03:28.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:28.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2021-05-12T09:03:28.284 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:28.284 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2021-05-12T09:03:28.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:28.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2021-05-12T09:03:28.484 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:28.484 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2021-05-12T09:03:28.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:28.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2021-05-12T09:03:28.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:28.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2021-05-12T09:03:28.794 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:28.794 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2021-05-12T09:03:28.955 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:28.955 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2021-05-12T09:03:29.080 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:29.081 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2021-05-12T09:03:29.179 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:29.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2021-05-12T09:03:29.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:29.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2021-05-12T09:03:29.415 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:29.415 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2021-05-12T09:03:29.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:29.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2021-05-12T09:03:29.615 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:29.615 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2021-05-12T09:03:29.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:29.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2021-05-12T09:03:29.850 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:29.850 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2021-05-12T09:03:29.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:29.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2021-05-12T09:03:30.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:30.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2021-05-12T09:03:30.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:30.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2021-05-12T09:03:30.286 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:30.286 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2021-05-12T09:03:30.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:30.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2021-05-12T09:03:30.487 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:30.487 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2021-05-12T09:03:30.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:30.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2021-05-12T09:03:30.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:30.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2021-05-12T09:03:30.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:30.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2021-05-12T09:03:30.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:30.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2021-05-12T09:03:31.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:31.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2021-05-12T09:03:31.165 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:31.166 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2021-05-12T09:03:31.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:31.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2021-05-12T09:03:31.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:31.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2021-05-12T09:03:31.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:31.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2021-05-12T09:03:31.660 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:31.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2021-05-12T09:03:31.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:31.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2021-05-12T09:03:31.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:31.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2021-05-12T09:03:31.995 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:31.996 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2021-05-12T09:03:32.104 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:32.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2021-05-12T09:03:32.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:32.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2021-05-12T09:03:32.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:32.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2021-05-12T09:03:32.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:32.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2021-05-12T09:03:32.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:32.549 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2021-05-12T09:03:32.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:32.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2021-05-12T09:03:32.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:32.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2021-05-12T09:03:32.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:32.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2021-05-12T09:03:33.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:33.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2021-05-12T09:03:33.135 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:33.135 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2021-05-12T09:03:33.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:33.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2021-05-12T09:03:33.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:33.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2021-05-12T09:03:33.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:33.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2021-05-12T09:03:33.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:33.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2021-05-12T09:03:33.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:33.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2021-05-12T09:03:33.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:33.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2021-05-12T09:03:33.964 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:33.965 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2021-05-12T09:03:34.073 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:34.073 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2021-05-12T09:03:34.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:34.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2021-05-12T09:03:34.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:34.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2021-05-12T09:03:34.425 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:34.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2021-05-12T09:03:34.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:34.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2021-05-12T09:03:34.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:34.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2021-05-12T09:03:34.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:34.811 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2021-05-12T09:03:34.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:34.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2021-05-12T09:03:35.071 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:35.071 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2021-05-12T09:03:35.171 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:35.171 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2021-05-12T09:03:35.305 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:35.305 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2021-05-12T09:03:35.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:35.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2021-05-12T09:03:35.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:35.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2021-05-12T09:03:35.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:35.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2021-05-12T09:03:35.807 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:35.808 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2021-05-12T09:03:35.941 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:35.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2021-05-12T09:03:36.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:36.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2021-05-12T09:03:36.176 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:36.176 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2021-05-12T09:03:36.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:36.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2021-05-12T09:03:36.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:36.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2021-05-12T09:03:36.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:36.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2021-05-12T09:03:36.695 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:36.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2021-05-12T09:03:36.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:36.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2021-05-12T09:03:36.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:36.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2021-05-12T09:03:37.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:37.065 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2021-05-12T09:03:37.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:37.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2021-05-12T09:03:37.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:37.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2021-05-12T09:03:37.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:37.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2021-05-12T09:03:37.550 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:37.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2021-05-12T09:03:37.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:37.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2021-05-12T09:03:37.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:37.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2021-05-12T09:03:37.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:37.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2021-05-12T09:03:38.094 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:38.094 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2021-05-12T09:03:38.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:38.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2021-05-12T09:03:38.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:38.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2021-05-12T09:03:38.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-05-12T09:03:38.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2021-05-12T09:03:38.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T09:03:38.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2021-05-12T09:03:38.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:03:38.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:03:38.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:03:38.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:03:38.583 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T09:03:38.583 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:03:38.583 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:03:38.583 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:03:38.584 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:03:38.584 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:03:38.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:03:38.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:03:38.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:03:38.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:03:38.639 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:03:39.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:03:39.036 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:03:39.036 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:03:39.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:03:39.037 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:03:39.037 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:03:39.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836496 2021-05-12T09:03:39.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836496 2021-05-12T09:03:39.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496' 2021-05-12T09:03:39.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:03:39.204 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:03:39.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672974 2021-05-12T09:03:39.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672974 2021-05-12T09:03:39.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496 1-42949672974' 2021-05-12T09:03:39.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:03:39.344 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:03:39.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509453 2021-05-12T09:03:39.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509453 2021-05-12T09:03:39.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509453' 2021-05-12T09:03:39.484 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:03:39.484 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2021-05-12T09:03:39.485 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:03:39.485 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:03:39.485 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836496 2021-05-12T09:03:39.486 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:03:39.486 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836496 2021-05-12T09:03:39.487 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-05-12T09:03:39.487 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836496 2021-05-12T09:03:39.487 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:03:39.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836494 -lt 21474836496 2021-05-12T09:03:39.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:03:40.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:03:40.876 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:03:41.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836496 -lt 21474836496 2021-05-12T09:03:41.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:03:41.277 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-05-12T09:03:41.277 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:03:41.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:03:41.279 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672974 2021-05-12T09:03:41.279 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:03:41.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672974 2021-05-12T09:03:41.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-05-12T09:03:41.280 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672974 2021-05-12T09:03:41.281 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:03:41.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672974 -lt 42949672974 2021-05-12T09:03:41.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:03:41.666 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509453 2021-05-12T09:03:41.666 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:03:41.667 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:03:41.668 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509453 2021-05-12T09:03:41.668 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:03:41.669 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509453 2021-05-12T09:03:41.669 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2021-05-12T09:03:41.670 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509453 2021-05-12T09:03:41.670 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:03:42.056 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509453 -lt 64424509453 2021-05-12T09:03:42.057 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:03:42.058 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:03:42.058 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:03:42.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:03:42.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:03:42.580 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:03:42.580 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:03:42.580 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:03:42.580 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:03:42.581 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:03:42.581 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:03:42.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T09:03:42.954 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:03:42.954 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:03:42.955 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:03:43.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T09:03:43.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:03:43.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:03:43.463 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2021-05-12T09:03:43.463 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local poolname=test 2021-05-12T09:03:43.464 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: local objectname=existing_1 2021-05-12T09:03:43.464 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: ceph --format json osd map test existing_1 2021-05-12T09:03:43.464 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: jq -r .pgid 2021-05-12T09:03:43.852 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=2.0 2021-05-12T09:03:43.853 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 1' 2021-05-12T09:03:43.853 INFO:tasks.workunit.client.0.smithi028.stdout:blackholing osds 0 1 2021-05-12T09:03:43.854 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2021-05-12T09:03:44.238 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:03:44.239 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-05-12T09:03:15.790700+0000 31'100 31:113 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:03:14.594281+0000 0'0 2021-05-12T09:03:14.594281+0000 0 2021-05-12T09:03:44.239 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:03:13.981969+0000 31'62 31:102 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:03:13.031503+0000 0'0 2021-05-12T09:03:13.031503+0000 0 2021-05-12T09:03:44.239 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:03:44.239 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:03:44.240 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:03:44.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2021-05-12T09:03:44.250 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2021-05-12T09:03:44.251 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T09:03:44.251 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T09:03:44.251 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:03:44.251 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:03:44.252 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:03:44.252 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.0.asok 2021-05-12T09:03:44.252 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2021-05-12T09:03:44.253 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.0.asok config set objectstore_blackhole 1 2021-05-12T09:03:44.347 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:03:44.347 INFO:tasks.workunit.client.0.smithi028.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T09:03:44.347 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:03:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2021-05-12T09:03:44.357 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.1 2021-05-12T09:03:44.357 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T09:03:44.357 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T09:03:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:03:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:03:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:03:44.358 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.1.asok 2021-05-12T09:03:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2021-05-12T09:03:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.1.asok config set objectstore_blackhole 1 2021-05-12T09:03:44.453 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:03:44.453 INFO:tasks.workunit.client.0.smithi028.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T09:03:44.453 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:03:44.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2021-05-12T09:03:44.462 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2021-05-12T09:03:44.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2021-05-12T09:03:44.463 INFO:tasks.workunit.client.0.smithi028.stdout:writing divergent object 2021-05-12T09:03:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2021-05-12T09:03:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2021-05-12T09:03:44.465 INFO:tasks.workunit.client.0.smithi028.stdout:create missing divergent object 2021-05-12T09:03:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2021-05-12T09:03:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2021-05-12T09:03:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: local pooltype=rep 2021-05-12T09:03:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: shift 2021-05-12T09:03:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: local which=data 2021-05-12T09:03:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: shift 2021-05-12T09:03:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: local poolname=test 2021-05-12T09:03:44.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: shift 2021-05-12T09:03:44.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: local objname=existing_99 2021-05-12T09:03:44.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: shift 2021-05-12T09:03:44.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: local dir=td/divergent-priors 2021-05-12T09:03:44.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: shift 2021-05-12T09:03:44.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: local shard_id=0 2021-05-12T09:03:44.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: shift 2021-05-12T09:03:44.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-05-12T09:03:44.471 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: get_osds test existing_99 2021-05-12T09:03:44.472 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local poolname=test 2021-05-12T09:03:44.472 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: get_osds: local objectname=existing_99 2021-05-12T09:03:44.472 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: ceph --format json osd map test existing_99 2021-05-12T09:03:44.472 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: jq '.acting | .[]' 2021-05-12T09:03:44.857 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local 'osds=2 2021-05-12T09:03:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:03:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:0' 2021-05-12T09:03:44.859 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: get_osds: echo 2 1 0 2021-05-12T09:03:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local -a initial_osds 2021-05-12T09:03:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: local osd_id=2 2021-05-12T09:03:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: '[' rep '!=' ec ']' 2021-05-12T09:03:44.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: shard_id= 2021-05-12T09:03:44.861 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: cat td/divergent-priors/2/type 2021-05-12T09:03:44.865 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: type=bluestore 2021-05-12T09:03:44.866 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2021-05-12T09:03:44.866 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local daemon=osd 2021-05-12T09:03:44.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local id=2 2021-05-12T09:03:44.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local config=bluestore_debug_inject_read_err 2021-05-12T09:03:44.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local value=true 2021-05-12T09:03:44.868 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: set_config: jq 'has("success")' 2021-05-12T09:03:44.868 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: set_config: get_asok_path osd.2 2021-05-12T09:03:44.869 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T09:03:44.869 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T09:03:44.869 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:03:44.870 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:03:44.870 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:03:44.870 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.2.asok 2021-05-12T09:03:44.870 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.29719/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2021-05-12T09:03:44.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: set_config: test true == true 2021-05-12T09:03:44.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: local loop=0 2021-05-12T09:03:44.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: inject_eio: grep -q Invalid 2021-05-12T09:03:44.968 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: get_asok_path osd.2 2021-05-12T09:03:44.968 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T09:03:44.969 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T09:03:44.969 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:03:44.969 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:03:44.969 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:03:44.970 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.2.asok 2021-05-12T09:03:44.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: CEPH_ARGS= 2021-05-12T09:03:44.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.2.asok injectdataerr test existing_99 2021-05-12T09:03:45.074 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2021-05-12T09:03:45.075 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2021-05-12T09:03:55.075 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2021-05-12T09:03:55.086 INFO:tasks.workunit.client.0.smithi028.stderr:rados: no process found 2021-05-12T09:03:55.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2021-05-12T09:03:55.087 INFO:tasks.workunit.client.0.smithi028.stdout:killing all the osds 2021-05-12T09:03:55.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2021-05-12T09:03:55.460 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:03:55.461 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 0 0 0 48300 0 0 17 17 active+clean 2021-05-12T09:03:48.609174+0000 31'101 31:122 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:03:14.594281+0000 0'0 2021-05-12T09:03:14.594281+0000 0 2021-05-12T09:03:55.461 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:03:13.981969+0000 31'62 31:102 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:03:13.031503+0000 0'0 2021-05-12T09:03:13.031503+0000 0 2021-05-12T09:03:55.462 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:03:55.462 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:03:55.463 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:03:55.472 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2021-05-12T09:03:55.473 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:03:55.474 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:03:55.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:03:55.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:03:55.477 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:03:55.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:03:55.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-05-12T09:03:55.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2021-05-12T09:03:56.105 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already down. 2021-05-12T09:03:56.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-05-12T09:03:56.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2021-05-12T09:03:56.637 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already down. 2021-05-12T09:03:56.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-05-12T09:03:56.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2021-05-12T09:03:57.180 INFO:tasks.workunit.client.0.smithi028.stderr:osd.2 is already down. 2021-05-12T09:03:57.190 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2021-05-12T09:03:57.190 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2021-05-12T09:03:57.713 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already out. 2021-05-12T09:03:57.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2021-05-12T09:03:57.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.1 2021-05-12T09:03:58.257 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already out. 2021-05-12T09:03:58.266 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 1' 2021-05-12T09:03:58.267 INFO:tasks.workunit.client.0.smithi028.stdout:bringing up non_divergent 0 1 2021-05-12T09:03:58.267 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2021-05-12T09:03:58.629 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:03:58.629 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 0 0 0 48300 0 0 17 17 stale+active+clean 2021-05-12T09:03:48.609174+0000 31'101 31:122 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:03:14.594281+0000 0'0 2021-05-12T09:03:14.594281+0000 0 2021-05-12T09:03:58.630 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 stale+active+clean 2021-05-12T09:03:13.981969+0000 31'62 31:102 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:03:13.031503+0000 0'0 2021-05-12T09:03:13.031503+0000 0 2021-05-12T09:03:58.630 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:03:58.630 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:03:58.631 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:03:58.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2021-05-12T09:03:58.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2021-05-12T09:03:58.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:03:58.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:03:58.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T09:03:58.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:03:58.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T09:03:58.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:03:58.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:03:58.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:03:58.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:03:58.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T09:03:58.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T09:03:58.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:03:58.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:03:58.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:03:58.648 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:03:58.648 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:03:58.648 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:03:58.648 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:03:58.649 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:03:58.649 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:03:58.649 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:03:58.649 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:03:58.649 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:03:58.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:03:58.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:03:58.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:03:58.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:03:58.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:03:58.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:03:58.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:03:58.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T09:03:58.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T09:03:58.652 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:03:58.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:03:58.654 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T09:03:58.654 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T09:03:58.654 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:03:58.654 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:03:58.655 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:03:58.673 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:58.673+0000 7f60e584e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:03:58.674 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:58.675+0000 7f60e584e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:03:58.675 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:58.675+0000 7f60e584e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:03:59.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T09:03:59.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:03:59.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:03:59.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:03:59.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:03:59.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:03:59.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:03:59.035 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:03:59.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:03:59.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:03:59.201 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:03:59.201+0000 7f60e584e0c0 -1 Falling back to public interface 2021-05-12T09:03:59.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:04:00.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:04:00.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:04:00.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:04:00.423 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:04:00.424 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:04:00.424 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:04:00.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:04:01.768 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:04:01.768+0000 7f60e584e0c0 -1 osd.0 31 log_to_monitors {default=true} 2021-05-12T09:04:01.900 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:04:01.901 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:04:01.902 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:04:01.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:04:01.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:04:01.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:04:02.216 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [5,31) [v2:127.0.0.1:6802/37667,v1:127.0.0.1:6803/37667] [v2:127.0.0.1:6804/37667,v1:127.0.0.1:6805/37667] exists,up 76b38569-e8bc-43f9-a984-1867943d6e8d 2021-05-12T09:04:02.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:04:02.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:04:02.221 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:04:02.221 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2021-05-12T09:04:02.221 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 1 2021-05-12T09:04:02.221 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:04:02.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:04:02.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T09:04:02.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:04:02.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T09:04:02.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:04:02.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:04:02.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:04:02.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:04:02.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T09:04:02.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T09:04:02.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:04:02.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:04:02.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:04:02.224 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:04:02.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:04:02.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:04:02.225 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:04:02.226 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:04:02.226 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:04:02.226 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:04:02.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:04:02.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:04:02.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:04:02.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:04:02.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:04:02.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:04:02.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:04:02.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:04:02.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:04:02.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T09:04:02.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T09:04:02.229 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:04:02.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:04:02.230 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T09:04:02.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T09:04:02.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:04:02.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:04:02.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:04:02.249 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:04:02.248+0000 7f62169bc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:04:02.277 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:04:02.277+0000 7f62169bc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:04:02.278 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:04:02.278+0000 7f62169bc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:04:02.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T09:04:02.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:04:02.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:04:02.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:04:02.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:04:02.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:04:02.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:04:02.609 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:04:02.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:04:02.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:04:02.798 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:04:02.798+0000 7f62169bc0c0 -1 Falling back to public interface 2021-05-12T09:04:03.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:04:04.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:04:04.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:04:04.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:04:04.042 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:04:04.042 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:04:04.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:04:04.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:04:05.351 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:04:05.351+0000 7f62169bc0c0 -1 osd.1 31 log_to_monitors {default=true} 2021-05-12T09:04:05.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:04:05.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:04:05.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:04:05.441 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:04:05.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:04:05.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:04:05.836 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up out weight 0 up_from 40 up_thru 20 down_at 33 last_clean_interval [10,31) [v2:127.0.0.1:6810/38232,v1:127.0.0.1:6811/38232] [v2:127.0.0.1:6812/38232,v1:127.0.0.1:6813/38232] exists,up fcbcbfab-6930-4442-a86e-6c4dd2cd2b5c 2021-05-12T09:04:05.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:04:05.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:04:05.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:04:05.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2021-05-12T09:04:05.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2021-05-12T09:04:06.346 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already in. 2021-05-12T09:04:06.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2021-05-12T09:04:06.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.1 2021-05-12T09:04:07.015 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already in. 2021-05-12T09:04:07.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T09:04:07.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2021-05-12T09:04:07.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:04:07.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:04:07.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:04:07.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:04:07.027 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T09:04:07.027 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:04:07.028 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:04:07.028 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:04:07.029 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:04:07.029 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:04:07.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:04:07.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:04:07.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:04:07.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:04:07.083 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:04:07.473 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:04:07.473 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:04:07.473 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:04:07.473 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:04:07.474 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:04:07.474 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:04:07.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=163208757252 2021-05-12T09:04:07.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 163208757252 2021-05-12T09:04:07.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-163208757252' 2021-05-12T09:04:07.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:04:07.663 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:04:07.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=171798691843 2021-05-12T09:04:07.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 171798691843 2021-05-12T09:04:07.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-163208757252 1-171798691843' 2021-05-12T09:04:07.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:04:07.804 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:04:07.908 INFO:tasks.workunit.client.0.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-05-12T09:04:07.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T09:04:07.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T09:04:07.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T09:04:07.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:04:07.911 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-163208757252 2021-05-12T09:04:07.911 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:04:07.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:04:07.912 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-163208757252 2021-05-12T09:04:07.912 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:04:07.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=163208757252 2021-05-12T09:04:07.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 163208757252' 2021-05-12T09:04:07.913 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 163208757252 2021-05-12T09:04:07.914 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:04:08.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 1 -lt 163208757252 2021-05-12T09:04:08.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:04:09.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:04:09.297 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:04:09.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 163208757252 -lt 163208757252 2021-05-12T09:04:09.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:04:09.695 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-171798691843 2021-05-12T09:04:09.695 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:04:09.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:04:09.696 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-171798691843 2021-05-12T09:04:09.696 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:04:09.697 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=171798691843 2021-05-12T09:04:09.697 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 171798691843' 2021-05-12T09:04:09.697 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 171798691843 2021-05-12T09:04:09.698 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:04:10.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 171798691843 -lt 171798691843 2021-05-12T09:04:10.105 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:04:10.106 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:04:10.106 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:04:10.615 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:04:10.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:04:10.616 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:04:10.616 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:04:10.616 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:04:10.617 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:04:10.617 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:04:10.617 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:04:10.988 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:04:10.989 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:04:10.989 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:04:10.989 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:04:11.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:04:11.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T09:04:11.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T09:04:11.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T09:04:11.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:04:11.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:04:11.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:04:11.597 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:04:11.597 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:04:11.597 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:04:11.597 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:04:11.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:04:11.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:04:11.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:04:11.974 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:04:11.974 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:04:11.975 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:04:11.975 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:04:12.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:04:12.482 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:04:12.482 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:04:12.482 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:04:12.482 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:04:12.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:04:12.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:04:12.483 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:04:12.483 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:04:12.988 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:04:12.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:04:12.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T09:04:12.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:04:12.990 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T09:04:13.190 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:04:13.190 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:04:13.190 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:04:13.191 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:04:13.191 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:04:13.191 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:04:13.191 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:04:13.191 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:04:13.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:04:13.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:04:13.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:04:13.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:04:14.072 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:04:14.072 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:04:14.073 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:04:14.073 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:04:14.073 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:04:14.073 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:04:14.074 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:04:14.074 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:04:14.074 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:04:14.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:04:14.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:04:14.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T09:04:14.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:04:14.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T09:04:14.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:04:14.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:04:14.984 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:04:14.984 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:04:14.985 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:04:14.985 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:04:14.985 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:04:14.985 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:04:15.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:04:15.373 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:04:15.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:04:15.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:04:15.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:04:15.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:04:15.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:04:15.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:04:15.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:04:15.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:04:15.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:04:15.883 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:04:15.884 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:04:16.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:04:16.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:04:16.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T09:04:16.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:04:16.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T09:04:17.190 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:04:17.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:04:17.192 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:04:17.192 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:04:17.192 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:04:17.192 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:04:17.193 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:04:17.193 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:04:17.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:04:17.582 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:04:17.582 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:04:17.583 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:04:18.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:04:18.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:04:18.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:04:18.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:04:18.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:04:18.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:04:18.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:04:18.091 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:04:18.091 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:04:18.595 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:04:18.595 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:04:18.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T09:04:18.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:04:18.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T09:04:20.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:04:20.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:04:20.197 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:04:20.198 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:04:20.198 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:04:20.198 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:04:20.199 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:04:20.199 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:04:20.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:04:20.585 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:04:20.585 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:04:20.586 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:04:21.095 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:04:21.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:04:21.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:04:21.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:04:21.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:04:21.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:04:21.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:04:21.097 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:04:21.097 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:04:21.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:04:21.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:04:21.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T09:04:21.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:04:21.600 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T09:04:24.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:04:24.800 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:04:24.801 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:04:24.801 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:04:24.801 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:04:24.801 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:04:24.801 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:04:24.802 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:04:25.189 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:04:25.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:04:25.190 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:04:25.190 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:04:25.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:04:25.730 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:04:25.730 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:04:25.730 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:04:25.730 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:04:25.731 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:04:25.731 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:04:25.731 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:04:25.731 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:04:26.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:04:26.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:04:26.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T09:04:26.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:04:26.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T09:04:32.634 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:04:32.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:04:32.636 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:04:32.636 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:04:32.636 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:04:32.637 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:04:32.637 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:04:32.637 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:04:33.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:04:33.025 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:04:33.025 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:04:33.026 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:04:33.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:04:33.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:04:33.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:04:33.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:04:33.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:04:33.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:04:33.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:04:33.541 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:04:33.541 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:04:34.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:04:34.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:04:34.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T09:04:34.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:04:34.049 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T09:04:41.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:04:41.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:04:41.349 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:04:41.349 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:04:41.349 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:04:41.349 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:04:41.350 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:04:41.350 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:04:41.737 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:04:41.737 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:04:41.737 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:04:41.738 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:04:42.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:04:42.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:04:42.246 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:04:42.246 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:04:42.247 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:04:42.247 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:04:42.247 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:04:42.247 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:04:42.247 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:04:42.759 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:04:42.760 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:04:42.760 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T09:04:42.760 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T09:04:43.280 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:04:43.281 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_fingerprint": "47e4c42a-8b42-4cd4-a357-4b2cd3623e93", 2021-05-12T09:04:43.281 INFO:tasks.workunit.client.0.smithi028.stdout: "version": "17.0.0-4095-ga8c673d5", 2021-05-12T09:04:43.281 INFO:tasks.workunit.client.0.smithi028.stdout: "commit": "a8c673d555488beec9ccda5812ba822e69c12ea9", 2021-05-12T09:04:43.281 INFO:tasks.workunit.client.0.smithi028.stdout: "timestamp": "2021-05-12T09:04:43.278548+0000", 2021-05-12T09:04:43.282 INFO:tasks.workunit.client.0.smithi028.stdout: "tag": "", 2021-05-12T09:04:43.282 INFO:tasks.workunit.client.0.smithi028.stdout: "health": { 2021-05-12T09:04:43.282 INFO:tasks.workunit.client.0.smithi028.stdout: "status": "HEALTH_WARN", 2021-05-12T09:04:43.282 INFO:tasks.workunit.client.0.smithi028.stdout: "checks": { 2021-05-12T09:04:43.282 INFO:tasks.workunit.client.0.smithi028.stdout: "OSDMAP_FLAGS": { 2021-05-12T09:04:43.283 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:04:43.283 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:04:43.283 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T09:04:43.283 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 17 2021-05-12T09:04:43.284 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.284 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [], 2021-05-12T09:04:43.284 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:04:43.284 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.284 INFO:tasks.workunit.client.0.smithi028.stdout: "OSD_DOWN": { 2021-05-12T09:04:43.285 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:04:43.285 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:04:43.285 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "1 osds down", 2021-05-12T09:04:43.285 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 1 2021-05-12T09:04:43.286 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.286 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:04:43.286 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.286 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "osd.2 (root=default,host=smithi028) is down" 2021-05-12T09:04:43.286 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.287 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.287 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:04:43.287 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.287 INFO:tasks.workunit.client.0.smithi028.stdout: "PG_DEGRADED": { 2021-05-12T09:04:43.288 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:04:43.288 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:04:43.288 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "Degraded data redundancy: 102/306 objects degraded (33.333%), 2 pgs degraded", 2021-05-12T09:04:43.288 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 2 2021-05-12T09:04:43.288 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.289 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:04:43.289 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.289 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:04:43.289 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.290 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.290 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.0 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:04:43.290 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.290 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.290 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:04:43.291 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.291 INFO:tasks.workunit.client.0.smithi028.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T09:04:43.291 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:04:43.291 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:04:43.291 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T09:04:43.292 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 1 2021-05-12T09:04:43.292 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.292 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:04:43.292 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.293 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T09:04:43.293 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.293 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.293 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T09:04:43.293 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.294 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.294 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:04:43.294 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.294 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.295 INFO:tasks.workunit.client.0.smithi028.stdout: "mutes": [] 2021-05-12T09:04:43.295 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.295 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap_first_committed": 1, 2021-05-12T09:04:43.295 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap_last_committed": 1, 2021-05-12T09:04:43.295 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap": { 2021-05-12T09:04:43.296 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 1, 2021-05-12T09:04:43.296 INFO:tasks.workunit.client.0.smithi028.stdout: "fsid": "8342ec9e-531f-44b4-8c8c-079d667fe908", 2021-05-12T09:04:43.296 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:02:42.842798Z", 2021-05-12T09:04:43.297 INFO:tasks.workunit.client.0.smithi028.stdout: "created": "2021-05-12T09:02:42.842798Z", 2021-05-12T09:04:43.297 INFO:tasks.workunit.client.0.smithi028.stdout: "min_mon_release": 17, 2021-05-12T09:04:43.297 INFO:tasks.workunit.client.0.smithi028.stdout: "min_mon_release_name": "quincy", 2021-05-12T09:04:43.297 INFO:tasks.workunit.client.0.smithi028.stdout: "election_strategy": 1, 2021-05-12T09:04:43.297 INFO:tasks.workunit.client.0.smithi028.stdout: "disallowed_leaders: ": "", 2021-05-12T09:04:43.298 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode": false, 2021-05-12T09:04:43.298 INFO:tasks.workunit.client.0.smithi028.stdout: "features": { 2021-05-12T09:04:43.298 INFO:tasks.workunit.client.0.smithi028.stdout: "persistent": [ 2021-05-12T09:04:43.298 INFO:tasks.workunit.client.0.smithi028.stdout: "kraken", 2021-05-12T09:04:43.299 INFO:tasks.workunit.client.0.smithi028.stdout: "luminous", 2021-05-12T09:04:43.299 INFO:tasks.workunit.client.0.smithi028.stdout: "mimic", 2021-05-12T09:04:43.299 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap-prune", 2021-05-12T09:04:43.299 INFO:tasks.workunit.client.0.smithi028.stdout: "nautilus", 2021-05-12T09:04:43.299 INFO:tasks.workunit.client.0.smithi028.stdout: "octopus", 2021-05-12T09:04:43.300 INFO:tasks.workunit.client.0.smithi028.stdout: "pacific", 2021-05-12T09:04:43.300 INFO:tasks.workunit.client.0.smithi028.stdout: "elector-pinging", 2021-05-12T09:04:43.300 INFO:tasks.workunit.client.0.smithi028.stdout: "quincy" 2021-05-12T09:04:43.300 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.301 INFO:tasks.workunit.client.0.smithi028.stdout: "optional": [] 2021-05-12T09:04:43.301 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.301 INFO:tasks.workunit.client.0.smithi028.stdout: "mons": [ 2021-05-12T09:04:43.301 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.302 INFO:tasks.workunit.client.0.smithi028.stdout: "rank": 0, 2021-05-12T09:04:43.302 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "a", 2021-05-12T09:04:43.302 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:04:43.302 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:04:43.303 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.303 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:04:43.303 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:7115", 2021-05-12T09:04:43.303 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 0 2021-05-12T09:04:43.303 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.304 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.304 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.304 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T09:04:43.304 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T09:04:43.305 INFO:tasks.workunit.client.0.smithi028.stdout: "priority": 0, 2021-05-12T09:04:43.305 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 0, 2021-05-12T09:04:43.305 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_location": "{}" 2021-05-12T09:04:43.305 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.305 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.306 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.306 INFO:tasks.workunit.client.0.smithi028.stdout: "quorum": [ 2021-05-12T09:04:43.306 INFO:tasks.workunit.client.0.smithi028.stdout: 0 2021-05-12T09:04:43.306 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.306 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap": { 2021-05-12T09:04:43.307 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 46, 2021-05-12T09:04:43.307 INFO:tasks.workunit.client.0.smithi028.stdout: "fsid": "8342ec9e-531f-44b4-8c8c-079d667fe908", 2021-05-12T09:04:43.307 INFO:tasks.workunit.client.0.smithi028.stdout: "created": "2021-05-12T09:02:43.641709+0000", 2021-05-12T09:04:43.307 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:04:07.575014+0000", 2021-05-12T09:04:43.308 INFO:tasks.workunit.client.0.smithi028.stdout: "last_up_change": "2021-05-12T09:04:05.612048+0000", 2021-05-12T09:04:43.308 INFO:tasks.workunit.client.0.smithi028.stdout: "last_in_change": "2021-05-12T09:04:06.694659+0000", 2021-05-12T09:04:43.308 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T09:04:43.308 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_num": 5800384, 2021-05-12T09:04:43.308 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_set": [ 2021-05-12T09:04:43.309 INFO:tasks.workunit.client.0.smithi028.stdout: "nodown", 2021-05-12T09:04:43.309 INFO:tasks.workunit.client.0.smithi028.stdout: "noin", 2021-05-12T09:04:43.309 INFO:tasks.workunit.client.0.smithi028.stdout: "noout", 2021-05-12T09:04:43.309 INFO:tasks.workunit.client.0.smithi028.stdout: "pglog_hardlimit", 2021-05-12T09:04:43.309 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snapdirs", 2021-05-12T09:04:43.310 INFO:tasks.workunit.client.0.smithi028.stdout: "recovery_deletes", 2021-05-12T09:04:43.310 INFO:tasks.workunit.client.0.smithi028.stdout: "sortbitwise" 2021-05-12T09:04:43.310 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.310 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_version": 9, 2021-05-12T09:04:43.311 INFO:tasks.workunit.client.0.smithi028.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T09:04:43.311 INFO:tasks.workunit.client.0.smithi028.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T09:04:43.311 INFO:tasks.workunit.client.0.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T09:04:43.311 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_snapshot": "", 2021-05-12T09:04:43.311 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_max": 2, 2021-05-12T09:04:43.312 INFO:tasks.workunit.client.0.smithi028.stdout: "max_osd": 3, 2021-05-12T09:04:43.312 INFO:tasks.workunit.client.0.smithi028.stdout: "require_min_compat_client": "luminous", 2021-05-12T09:04:43.312 INFO:tasks.workunit.client.0.smithi028.stdout: "min_compat_client": "jewel", 2021-05-12T09:04:43.312 INFO:tasks.workunit.client.0.smithi028.stdout: "require_osd_release": "quincy", 2021-05-12T09:04:43.313 INFO:tasks.workunit.client.0.smithi028.stdout: "pools": [ 2021-05-12T09:04:43.313 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.313 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 1, 2021-05-12T09:04:43.313 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_name": ".mgr", 2021-05-12T09:04:43.313 INFO:tasks.workunit.client.0.smithi028.stdout: "create_time": "2021-05-12T09:03:12.994391+0000", 2021-05-12T09:04:43.314 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": 1, 2021-05-12T09:04:43.314 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_names": "hashpspool", 2021-05-12T09:04:43.314 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:04:43.314 INFO:tasks.workunit.client.0.smithi028.stdout: "size": 3, 2021-05-12T09:04:43.315 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 2, 2021-05-12T09:04:43.315 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_rule": 0, 2021-05-12T09:04:43.315 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_count": 0, 2021-05-12T09:04:43.315 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_target": 0, 2021-05-12T09:04:43.315 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T09:04:43.316 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T09:04:43.316 INFO:tasks.workunit.client.0.smithi028.stdout: "object_hash": 2, 2021-05-12T09:04:43.316 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_autoscale_mode": "off", 2021-05-12T09:04:43.316 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num": 1, 2021-05-12T09:04:43.316 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num": 1, 2021-05-12T09:04:43.317 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num_target": 1, 2021-05-12T09:04:43.317 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_target": 1, 2021-05-12T09:04:43.317 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_pending": 1, 2021-05-12T09:04:43.317 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pg_merge_meta": { 2021-05-12T09:04:43.318 INFO:tasks.workunit.client.0.smithi028.stdout: "source_pgid": "0.0", 2021-05-12T09:04:43.318 INFO:tasks.workunit.client.0.smithi028.stdout: "ready_epoch": 0, 2021-05-12T09:04:43.318 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_started": 0, 2021-05-12T09:04:43.318 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": 0, 2021-05-12T09:04:43.318 INFO:tasks.workunit.client.0.smithi028.stdout: "source_version": "0'0", 2021-05-12T09:04:43.319 INFO:tasks.workunit.client.0.smithi028.stdout: "target_version": "0'0" 2021-05-12T09:04:43.319 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.319 INFO:tasks.workunit.client.0.smithi028.stdout: "last_change": "24", 2021-05-12T09:04:43.319 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend": "0", 2021-05-12T09:04:43.320 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T09:04:43.320 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T09:04:43.320 INFO:tasks.workunit.client.0.smithi028.stdout: "auid": 0, 2021-05-12T09:04:43.320 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_mode": "selfmanaged", 2021-05-12T09:04:43.320 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_seq": 0, 2021-05-12T09:04:43.321 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_epoch": 0, 2021-05-12T09:04:43.321 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_snaps": [], 2021-05-12T09:04:43.321 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps": "[]", 2021-05-12T09:04:43.321 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_bytes": 0, 2021-05-12T09:04:43.321 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_objects": 0, 2021-05-12T09:04:43.322 INFO:tasks.workunit.client.0.smithi028.stdout: "tiers": [], 2021-05-12T09:04:43.322 INFO:tasks.workunit.client.0.smithi028.stdout: "tier_of": -1, 2021-05-12T09:04:43.322 INFO:tasks.workunit.client.0.smithi028.stdout: "read_tier": -1, 2021-05-12T09:04:43.322 INFO:tasks.workunit.client.0.smithi028.stdout: "write_tier": -1, 2021-05-12T09:04:43.323 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_mode": "none", 2021-05-12T09:04:43.323 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_bytes": 0, 2021-05-12T09:04:43.323 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_objects": 0, 2021-05-12T09:04:43.323 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T09:04:43.323 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T09:04:43.324 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T09:04:43.324 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_flush_age": 0, 2021-05-12T09:04:43.324 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_evict_age": 0, 2021-05-12T09:04:43.324 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profile": "", 2021-05-12T09:04:43.324 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_params": { 2021-05-12T09:04:43.325 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "none" 2021-05-12T09:04:43.325 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.325 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_period": 0, 2021-05-12T09:04:43.325 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_count": 0, 2021-05-12T09:04:43.326 INFO:tasks.workunit.client.0.smithi028.stdout: "use_gmt_hitset": true, 2021-05-12T09:04:43.326 INFO:tasks.workunit.client.0.smithi028.stdout: "min_read_recency_for_promote": 0, 2021-05-12T09:04:43.326 INFO:tasks.workunit.client.0.smithi028.stdout: "min_write_recency_for_promote": 0, 2021-05-12T09:04:43.326 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T09:04:43.326 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_search_last_n": 0, 2021-05-12T09:04:43.327 INFO:tasks.workunit.client.0.smithi028.stdout: "grade_table": [], 2021-05-12T09:04:43.327 INFO:tasks.workunit.client.0.smithi028.stdout: "stripe_width": 0, 2021-05-12T09:04:43.327 INFO:tasks.workunit.client.0.smithi028.stdout: "expected_num_objects": 0, 2021-05-12T09:04:43.327 INFO:tasks.workunit.client.0.smithi028.stdout: "fast_read": false, 2021-05-12T09:04:43.327 INFO:tasks.workunit.client.0.smithi028.stdout: "options": { 2021-05-12T09:04:43.328 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_min": 1 2021-05-12T09:04:43.328 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.328 INFO:tasks.workunit.client.0.smithi028.stdout: "application_metadata": { 2021-05-12T09:04:43.328 INFO:tasks.workunit.client.0.smithi028.stdout: "mgr": {} 2021-05-12T09:04:43.329 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.329 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.329 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.329 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 2, 2021-05-12T09:04:43.329 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_name": "test", 2021-05-12T09:04:43.330 INFO:tasks.workunit.client.0.smithi028.stdout: "create_time": "2021-05-12T09:03:14.544175+0000", 2021-05-12T09:04:43.330 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": 1, 2021-05-12T09:04:43.330 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_names": "hashpspool", 2021-05-12T09:04:43.330 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:04:43.330 INFO:tasks.workunit.client.0.smithi028.stdout: "size": 3, 2021-05-12T09:04:43.331 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 2, 2021-05-12T09:04:43.331 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_rule": 0, 2021-05-12T09:04:43.331 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_count": 0, 2021-05-12T09:04:43.331 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_target": 0, 2021-05-12T09:04:43.332 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T09:04:43.332 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T09:04:43.332 INFO:tasks.workunit.client.0.smithi028.stdout: "object_hash": 2, 2021-05-12T09:04:43.332 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_autoscale_mode": "off", 2021-05-12T09:04:43.332 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num": 1, 2021-05-12T09:04:43.333 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num": 1, 2021-05-12T09:04:43.333 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num_target": 1, 2021-05-12T09:04:43.333 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_target": 1, 2021-05-12T09:04:43.333 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_pending": 1, 2021-05-12T09:04:43.334 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pg_merge_meta": { 2021-05-12T09:04:43.334 INFO:tasks.workunit.client.0.smithi028.stdout: "source_pgid": "0.0", 2021-05-12T09:04:43.334 INFO:tasks.workunit.client.0.smithi028.stdout: "ready_epoch": 0, 2021-05-12T09:04:43.334 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_started": 0, 2021-05-12T09:04:43.334 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": 0, 2021-05-12T09:04:43.335 INFO:tasks.workunit.client.0.smithi028.stdout: "source_version": "0'0", 2021-05-12T09:04:43.335 INFO:tasks.workunit.client.0.smithi028.stdout: "target_version": "0'0" 2021-05-12T09:04:43.335 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.335 INFO:tasks.workunit.client.0.smithi028.stdout: "last_change": "31", 2021-05-12T09:04:43.335 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend": "0", 2021-05-12T09:04:43.336 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T09:04:43.336 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T09:04:43.336 INFO:tasks.workunit.client.0.smithi028.stdout: "auid": 0, 2021-05-12T09:04:43.336 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_mode": "selfmanaged", 2021-05-12T09:04:43.337 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_seq": 0, 2021-05-12T09:04:43.337 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_epoch": 0, 2021-05-12T09:04:43.337 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_snaps": [], 2021-05-12T09:04:43.337 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps": "[]", 2021-05-12T09:04:43.337 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_bytes": 0, 2021-05-12T09:04:43.338 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_objects": 0, 2021-05-12T09:04:43.338 INFO:tasks.workunit.client.0.smithi028.stdout: "tiers": [], 2021-05-12T09:04:43.338 INFO:tasks.workunit.client.0.smithi028.stdout: "tier_of": -1, 2021-05-12T09:04:43.338 INFO:tasks.workunit.client.0.smithi028.stdout: "read_tier": -1, 2021-05-12T09:04:43.339 INFO:tasks.workunit.client.0.smithi028.stdout: "write_tier": -1, 2021-05-12T09:04:43.339 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_mode": "none", 2021-05-12T09:04:43.339 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_bytes": 0, 2021-05-12T09:04:43.339 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_objects": 0, 2021-05-12T09:04:43.339 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T09:04:43.340 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T09:04:43.340 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T09:04:43.340 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_flush_age": 0, 2021-05-12T09:04:43.340 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_evict_age": 0, 2021-05-12T09:04:43.340 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profile": "", 2021-05-12T09:04:43.341 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_params": { 2021-05-12T09:04:43.341 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "none" 2021-05-12T09:04:43.341 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.341 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_period": 0, 2021-05-12T09:04:43.342 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_count": 0, 2021-05-12T09:04:43.342 INFO:tasks.workunit.client.0.smithi028.stdout: "use_gmt_hitset": true, 2021-05-12T09:04:43.342 INFO:tasks.workunit.client.0.smithi028.stdout: "min_read_recency_for_promote": 0, 2021-05-12T09:04:43.342 INFO:tasks.workunit.client.0.smithi028.stdout: "min_write_recency_for_promote": 0, 2021-05-12T09:04:43.342 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T09:04:43.343 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_search_last_n": 0, 2021-05-12T09:04:43.343 INFO:tasks.workunit.client.0.smithi028.stdout: "grade_table": [], 2021-05-12T09:04:43.343 INFO:tasks.workunit.client.0.smithi028.stdout: "stripe_width": 0, 2021-05-12T09:04:43.343 INFO:tasks.workunit.client.0.smithi028.stdout: "expected_num_objects": 0, 2021-05-12T09:04:43.344 INFO:tasks.workunit.client.0.smithi028.stdout: "fast_read": false, 2021-05-12T09:04:43.344 INFO:tasks.workunit.client.0.smithi028.stdout: "options": {}, 2021-05-12T09:04:43.344 INFO:tasks.workunit.client.0.smithi028.stdout: "application_metadata": {} 2021-05-12T09:04:43.344 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.344 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.345 INFO:tasks.workunit.client.0.smithi028.stdout: "osds": [ 2021-05-12T09:04:43.345 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.345 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:04:43.345 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "76b38569-e8bc-43f9-a984-1867943d6e8d", 2021-05-12T09:04:43.345 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 1, 2021-05-12T09:04:43.346 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:04:43.346 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:04:43.346 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:04:43.346 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 5, 2021-05-12T09:04:43.347 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 31, 2021-05-12T09:04:43.347 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 38, 2021-05-12T09:04:43.347 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 43, 2021-05-12T09:04:43.347 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 32, 2021-05-12T09:04:43.347 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:04:43.348 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:04:43.348 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:04:43.348 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.348 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:04:43.348 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6802", 2021-05-12T09:04:43.349 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 37667 2021-05-12T09:04:43.349 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.349 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.349 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:04:43.350 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6803", 2021-05-12T09:04:43.350 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 37667 2021-05-12T09:04:43.350 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.350 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.350 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.351 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:04:43.351 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:04:43.351 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.351 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:04:43.352 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6804", 2021-05-12T09:04:43.352 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 37667 2021-05-12T09:04:43.352 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.352 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.352 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:04:43.353 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6805", 2021-05-12T09:04:43.353 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 37667 2021-05-12T09:04:43.353 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.353 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.353 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.354 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:04:43.354 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:04:43.354 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.354 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:04:43.355 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6808", 2021-05-12T09:04:43.355 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 37667 2021-05-12T09:04:43.355 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.355 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.355 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:04:43.356 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6809", 2021-05-12T09:04:43.356 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 37667 2021-05-12T09:04:43.356 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.356 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.357 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.357 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:04:43.357 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:04:43.357 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.357 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:04:43.358 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6806", 2021-05-12T09:04:43.358 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 37667 2021-05-12T09:04:43.358 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.358 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.358 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:04:43.359 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6807", 2021-05-12T09:04:43.359 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 37667 2021-05-12T09:04:43.359 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.359 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.360 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.360 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6803/37667", 2021-05-12T09:04:43.360 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6805/37667", 2021-05-12T09:04:43.360 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6809/37667", 2021-05-12T09:04:43.360 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6807/37667", 2021-05-12T09:04:43.361 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:04:43.361 INFO:tasks.workunit.client.0.smithi028.stdout: "exists", 2021-05-12T09:04:43.361 INFO:tasks.workunit.client.0.smithi028.stdout: "up" 2021-05-12T09:04:43.361 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.361 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.362 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.362 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:04:43.362 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "fcbcbfab-6930-4442-a86e-6c4dd2cd2b5c", 2021-05-12T09:04:43.362 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 1, 2021-05-12T09:04:43.363 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:04:43.363 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:04:43.363 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:04:43.363 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 10, 2021-05-12T09:04:43.364 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 31, 2021-05-12T09:04:43.364 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 40, 2021-05-12T09:04:43.364 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 45, 2021-05-12T09:04:43.364 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 33, 2021-05-12T09:04:43.364 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:04:43.365 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:04:43.365 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:04:43.365 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.366 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:04:43.366 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6810", 2021-05-12T09:04:43.366 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 38232 2021-05-12T09:04:43.366 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.366 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.367 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:04:43.367 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6811", 2021-05-12T09:04:43.367 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 38232 2021-05-12T09:04:43.367 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.368 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.368 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.368 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:04:43.368 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:04:43.368 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.369 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:04:43.369 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6812", 2021-05-12T09:04:43.369 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 38232 2021-05-12T09:04:43.369 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.369 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.370 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:04:43.370 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6813", 2021-05-12T09:04:43.370 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 38232 2021-05-12T09:04:43.370 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.371 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.371 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.371 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:04:43.371 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:04:43.371 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.372 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:04:43.372 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6816", 2021-05-12T09:04:43.372 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 38232 2021-05-12T09:04:43.372 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.372 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.373 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:04:43.373 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6817", 2021-05-12T09:04:43.373 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 38232 2021-05-12T09:04:43.373 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.374 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.374 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.374 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:04:43.374 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:04:43.375 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.375 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:04:43.375 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6814", 2021-05-12T09:04:43.375 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 38232 2021-05-12T09:04:43.375 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.376 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.376 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:04:43.376 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6815", 2021-05-12T09:04:43.376 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 38232 2021-05-12T09:04:43.376 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.377 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.377 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.377 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6811/38232", 2021-05-12T09:04:43.377 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6813/38232", 2021-05-12T09:04:43.377 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6817/38232", 2021-05-12T09:04:43.378 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6815/38232", 2021-05-12T09:04:43.378 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:04:43.378 INFO:tasks.workunit.client.0.smithi028.stdout: "exists", 2021-05-12T09:04:43.378 INFO:tasks.workunit.client.0.smithi028.stdout: "up" 2021-05-12T09:04:43.379 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.379 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.379 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.379 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:04:43.379 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "786b0ee9-ebdd-46c7-9baf-4db98735ea54", 2021-05-12T09:04:43.380 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 0, 2021-05-12T09:04:43.380 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:04:43.380 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:04:43.380 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:04:43.380 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 0, 2021-05-12T09:04:43.381 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 0, 2021-05-12T09:04:43.381 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 15, 2021-05-12T09:04:43.381 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 25, 2021-05-12T09:04:43.381 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 34, 2021-05-12T09:04:43.382 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:04:43.382 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:04:43.382 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:04:43.382 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.382 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:04:43.383 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6818", 2021-05-12T09:04:43.383 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 33624 2021-05-12T09:04:43.383 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.383 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.384 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:04:43.384 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6819", 2021-05-12T09:04:43.384 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 33624 2021-05-12T09:04:43.384 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.384 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.385 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.385 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:04:43.385 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:04:43.385 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.385 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:04:43.386 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6820", 2021-05-12T09:04:43.386 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 33624 2021-05-12T09:04:43.386 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.386 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.387 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:04:43.387 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6821", 2021-05-12T09:04:43.387 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 33624 2021-05-12T09:04:43.387 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.387 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.388 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.388 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:04:43.388 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:04:43.388 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.389 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:04:43.389 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6824", 2021-05-12T09:04:43.389 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 33624 2021-05-12T09:04:43.389 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.389 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.390 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:04:43.390 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6825", 2021-05-12T09:04:43.390 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 33624 2021-05-12T09:04:43.390 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.390 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.391 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.391 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:04:43.391 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:04:43.391 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.391 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:04:43.392 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6822", 2021-05-12T09:04:43.392 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 33624 2021-05-12T09:04:43.392 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.392 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.393 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:04:43.393 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6823", 2021-05-12T09:04:43.393 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 33624 2021-05-12T09:04:43.393 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.393 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.394 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.394 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6819/33624", 2021-05-12T09:04:43.394 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6821/33624", 2021-05-12T09:04:43.394 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6825/33624", 2021-05-12T09:04:43.394 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6823/33624", 2021-05-12T09:04:43.395 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:04:43.395 INFO:tasks.workunit.client.0.smithi028.stdout: "exists" 2021-05-12T09:04:43.395 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.395 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.396 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.396 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_xinfo": [ 2021-05-12T09:04:43.396 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.396 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:04:43.396 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:03:55.909790+0000", 2021-05-12T09:04:43.397 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:04:43.397 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:04:43.397 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:04:43.397 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:04:43.398 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:02:52.846962+0000", 2021-05-12T09:04:43.398 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:04:43.398 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.398 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.398 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:04:43.399 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:03:56.442468+0000", 2021-05-12T09:04:43.399 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:04:43.399 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:04:43.399 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:04:43.400 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:04:43.400 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:03:02.545287+0000", 2021-05-12T09:04:43.400 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:04:43.400 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.400 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.401 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:04:43.401 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:03:56.975757+0000", 2021-05-12T09:04:43.401 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:04:43.401 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:04:43.402 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:04:43.402 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:04:43.402 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:03:11.815937+0000", 2021-05-12T09:04:43.402 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:04:43.402 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.403 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.403 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_upmap": [], 2021-05-12T09:04:43.403 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_upmap_items": [], 2021-05-12T09:04:43.403 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_temp": [], 2021-05-12T09:04:43.404 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_temp": [], 2021-05-12T09:04:43.404 INFO:tasks.workunit.client.0.smithi028.stdout: "blocklist": {}, 2021-05-12T09:04:43.404 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profiles": { 2021-05-12T09:04:43.404 INFO:tasks.workunit.client.0.smithi028.stdout: "default": { 2021-05-12T09:04:43.404 INFO:tasks.workunit.client.0.smithi028.stdout: "k": "2", 2021-05-12T09:04:43.405 INFO:tasks.workunit.client.0.smithi028.stdout: "m": "2", 2021-05-12T09:04:43.405 INFO:tasks.workunit.client.0.smithi028.stdout: "plugin": "jerasure", 2021-05-12T09:04:43.405 INFO:tasks.workunit.client.0.smithi028.stdout: "technique": "reed_sol_van" 2021-05-12T09:04:43.405 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.405 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.406 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps_queue": [], 2021-05-12T09:04:43.406 INFO:tasks.workunit.client.0.smithi028.stdout: "new_removed_snaps": [], 2021-05-12T09:04:43.406 INFO:tasks.workunit.client.0.smithi028.stdout: "new_purged_snaps": [], 2021-05-12T09:04:43.406 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_node_flags": {}, 2021-05-12T09:04:43.407 INFO:tasks.workunit.client.0.smithi028.stdout: "device_class_flags": {}, 2021-05-12T09:04:43.407 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode": { 2021-05-12T09:04:43.407 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode_enabled": false, 2021-05-12T09:04:43.407 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_bucket_count": 0, 2021-05-12T09:04:43.407 INFO:tasks.workunit.client.0.smithi028.stdout: "degraded_stretch_mode": 0, 2021-05-12T09:04:43.408 INFO:tasks.workunit.client.0.smithi028.stdout: "recovering_stretch_mode": 0, 2021-05-12T09:04:43.408 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode_bucket": 0 2021-05-12T09:04:43.408 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.408 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.408 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_metadata": [ 2021-05-12T09:04:43.409 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.409 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:04:43.409 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:04:43.409 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6804/37667,v1:127.0.0.1:6805/37667]", 2021-05-12T09:04:43.410 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:04:43.410 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:04:43.410 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:04:43.410 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:04:43.410 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:04:43.411 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:04:43.411 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:04:43.411 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:04:43.411 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:04:43.411 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T09:04:43.412 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:04:43.412 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:04:43.412 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:04:43.412 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:04:43.413 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:04:43.413 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:04:43.413 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:04:43.413 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:04:43.413 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:04:43.414 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:04:43.414 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:04:43.414 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:04:43.414 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:04:43.415 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:04:43.415 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:04:43.415 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6802/37667,v1:127.0.0.1:6803/37667]", 2021-05-12T09:04:43.415 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:04:43.415 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/37667,v1:127.0.0.1:6809/37667]", 2021-05-12T09:04:43.416 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/37667,v1:127.0.0.1:6807/37667]", 2021-05-12T09:04:43.416 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:04:43.416 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:04:43.416 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:04:43.416 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:04:43.417 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:04:43.417 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:04:43.417 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:04:43.417 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:04:43.417 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:04:43.418 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T09:04:43.418 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:04:43.418 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:04:43.418 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:04:43.419 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.419 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.419 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:04:43.419 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:04:43.419 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6812/38232,v1:127.0.0.1:6813/38232]", 2021-05-12T09:04:43.420 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:04:43.420 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:04:43.420 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:04:43.420 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:04:43.421 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:04:43.421 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:04:43.422 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:04:43.422 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:04:43.422 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:04:43.422 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T09:04:43.423 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:04:43.423 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:04:43.423 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:04:43.423 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:04:43.424 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:04:43.424 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:04:43.424 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:04:43.424 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:04:43.424 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:04:43.425 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:04:43.425 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:04:43.425 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:04:43.425 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:04:43.425 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:04:43.426 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:04:43.426 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6810/38232,v1:127.0.0.1:6811/38232]", 2021-05-12T09:04:43.426 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:04:43.427 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/38232,v1:127.0.0.1:6817/38232]", 2021-05-12T09:04:43.427 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/38232,v1:127.0.0.1:6815/38232]", 2021-05-12T09:04:43.427 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:04:43.427 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:04:43.427 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:04:43.428 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:04:43.428 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:04:43.428 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:04:43.428 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:04:43.429 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:04:43.429 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:04:43.429 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T09:04:43.429 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:04:43.429 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:04:43.430 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:04:43.430 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.430 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.430 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:04:43.431 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:04:43.431 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6820/33624,v1:127.0.0.1:6821/33624]", 2021-05-12T09:04:43.431 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:04:43.431 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:04:43.431 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:04:43.432 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:04:43.432 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:04:43.432 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:04:43.432 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:04:43.432 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:04:43.433 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:04:43.433 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T09:04:43.433 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:04:43.433 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:04:43.434 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:04:43.434 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:04:43.434 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:04:43.434 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:04:43.434 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:04:43.435 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:04:43.435 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:04:43.435 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:04:43.435 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:04:43.435 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:04:43.436 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:04:43.436 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:04:43.436 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:04:43.436 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6818/33624,v1:127.0.0.1:6819/33624]", 2021-05-12T09:04:43.437 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:04:43.437 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/33624,v1:127.0.0.1:6825/33624]", 2021-05-12T09:04:43.437 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/33624,v1:127.0.0.1:6823/33624]", 2021-05-12T09:04:43.437 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:04:43.437 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:04:43.438 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:04:43.438 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:04:43.438 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:04:43.438 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:04:43.439 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:04:43.439 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:04:43.439 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:04:43.439 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T09:04:43.439 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:04:43.440 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:04:43.440 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:04:43.440 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.440 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.440 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_clean_epochs": { 2021-05-12T09:04:43.441 INFO:tasks.workunit.client.0.smithi028.stdout: "min_last_epoch_clean": 0, 2021-05-12T09:04:43.441 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": { 2021-05-12T09:04:43.441 INFO:tasks.workunit.client.0.smithi028.stdout: "per_pool": [] 2021-05-12T09:04:43.441 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.442 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_epochs": [ 2021-05-12T09:04:43.442 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.442 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:04:43.442 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 38 2021-05-12T09:04:43.442 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.443 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.443 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:04:43.443 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 41 2021-05-12T09:04:43.443 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.444 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.444 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:04:43.444 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 15 2021-05-12T09:04:43.444 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.444 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.445 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.445 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_first_committed": 1, 2021-05-12T09:04:43.445 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_last_committed": 46, 2021-05-12T09:04:43.445 INFO:tasks.workunit.client.0.smithi028.stdout: "crushmap": { 2021-05-12T09:04:43.446 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": [ 2021-05-12T09:04:43.446 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.446 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:04:43.446 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.0", 2021-05-12T09:04:43.446 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:04:43.447 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.447 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.447 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:04:43.447 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.1", 2021-05-12T09:04:43.448 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:04:43.448 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.448 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.448 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:04:43.448 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.2", 2021-05-12T09:04:43.449 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:04:43.449 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.449 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.449 INFO:tasks.workunit.client.0.smithi028.stdout: "types": [ 2021-05-12T09:04:43.450 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.450 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 0, 2021-05-12T09:04:43.450 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd" 2021-05-12T09:04:43.450 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.450 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.451 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:04:43.451 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "host" 2021-05-12T09:04:43.451 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.451 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.451 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 2, 2021-05-12T09:04:43.452 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "chassis" 2021-05-12T09:04:43.452 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.452 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.452 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 3, 2021-05-12T09:04:43.453 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "rack" 2021-05-12T09:04:43.453 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.453 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.453 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 4, 2021-05-12T09:04:43.453 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "row" 2021-05-12T09:04:43.454 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.454 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.454 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 5, 2021-05-12T09:04:43.454 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "pdu" 2021-05-12T09:04:43.454 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.455 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.455 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 6, 2021-05-12T09:04:43.455 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "pod" 2021-05-12T09:04:43.455 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.456 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.456 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 7, 2021-05-12T09:04:43.456 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "room" 2021-05-12T09:04:43.456 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.456 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.457 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 8, 2021-05-12T09:04:43.457 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "datacenter" 2021-05-12T09:04:43.457 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.457 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.457 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 9, 2021-05-12T09:04:43.458 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "zone" 2021-05-12T09:04:43.458 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.458 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.458 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 10, 2021-05-12T09:04:43.459 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "region" 2021-05-12T09:04:43.459 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.459 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.459 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:04:43.459 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "root" 2021-05-12T09:04:43.460 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.460 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.460 INFO:tasks.workunit.client.0.smithi028.stdout: "buckets": [ 2021-05-12T09:04:43.460 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.460 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -1, 2021-05-12T09:04:43.461 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "default", 2021-05-12T09:04:43.461 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:04:43.461 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "root", 2021-05-12T09:04:43.461 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:04:43.462 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:04:43.462 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:04:43.462 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:04:43.462 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.462 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -3, 2021-05-12T09:04:43.463 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:04:43.463 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:04:43.463 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.463 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.463 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.464 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.464 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -2, 2021-05-12T09:04:43.464 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "default~hdd", 2021-05-12T09:04:43.464 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:04:43.465 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "root", 2021-05-12T09:04:43.465 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:04:43.465 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:04:43.465 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:04:43.465 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:04:43.466 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.466 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -4, 2021-05-12T09:04:43.466 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:04:43.466 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:04:43.466 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.467 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.467 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.467 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.467 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -3, 2021-05-12T09:04:43.468 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "smithi028", 2021-05-12T09:04:43.468 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:04:43.468 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "host", 2021-05-12T09:04:43.468 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:04:43.468 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:04:43.469 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:04:43.469 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:04:43.469 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.469 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:04:43.469 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:04:43.470 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:04:43.470 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.470 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.470 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:04:43.471 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:04:43.471 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 1 2021-05-12T09:04:43.471 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.471 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.471 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:04:43.472 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:04:43.472 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 2 2021-05-12T09:04:43.472 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.472 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.473 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.473 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.473 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -4, 2021-05-12T09:04:43.473 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "smithi028~hdd", 2021-05-12T09:04:43.473 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:04:43.474 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "host", 2021-05-12T09:04:43.474 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:04:43.474 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:04:43.474 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:04:43.474 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:04:43.475 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.475 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:04:43.475 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:04:43.475 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:04:43.476 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.476 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.476 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:04:43.476 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:04:43.476 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 1 2021-05-12T09:04:43.477 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.477 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.477 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:04:43.477 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:04:43.477 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 2 2021-05-12T09:04:43.478 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.478 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.478 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.478 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.479 INFO:tasks.workunit.client.0.smithi028.stdout: "rules": [ 2021-05-12T09:04:43.480 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.480 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_id": 0, 2021-05-12T09:04:43.480 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_name": "replicated_rule", 2021-05-12T09:04:43.480 INFO:tasks.workunit.client.0.smithi028.stdout: "ruleset": 0, 2021-05-12T09:04:43.481 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:04:43.481 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 1, 2021-05-12T09:04:43.481 INFO:tasks.workunit.client.0.smithi028.stdout: "max_size": 10, 2021-05-12T09:04:43.481 INFO:tasks.workunit.client.0.smithi028.stdout: "steps": [ 2021-05-12T09:04:43.482 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.482 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "take", 2021-05-12T09:04:43.482 INFO:tasks.workunit.client.0.smithi028.stdout: "item": -1, 2021-05-12T09:04:43.482 INFO:tasks.workunit.client.0.smithi028.stdout: "item_name": "default" 2021-05-12T09:04:43.482 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.483 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.484 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "choose_firstn", 2021-05-12T09:04:43.484 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 0, 2021-05-12T09:04:43.484 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "osd" 2021-05-12T09:04:43.484 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.485 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.485 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "emit" 2021-05-12T09:04:43.485 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.485 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:04:43.486 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.486 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.486 INFO:tasks.workunit.client.0.smithi028.stdout: "tunables": { 2021-05-12T09:04:43.486 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_local_tries": 0, 2021-05-12T09:04:43.487 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_local_fallback_tries": 0, 2021-05-12T09:04:43.487 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_total_tries": 50, 2021-05-12T09:04:43.487 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_descend_once": 1, 2021-05-12T09:04:43.487 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_vary_r": 1, 2021-05-12T09:04:43.487 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_stable": 1, 2021-05-12T09:04:43.488 INFO:tasks.workunit.client.0.smithi028.stdout: "straw_calc_version": 1, 2021-05-12T09:04:43.488 INFO:tasks.workunit.client.0.smithi028.stdout: "allowed_bucket_algs": 54, 2021-05-12T09:04:43.488 INFO:tasks.workunit.client.0.smithi028.stdout: "profile": "jewel", 2021-05-12T09:04:43.488 INFO:tasks.workunit.client.0.smithi028.stdout: "optimal_tunables": 1, 2021-05-12T09:04:43.489 INFO:tasks.workunit.client.0.smithi028.stdout: "legacy_tunables": 0, 2021-05-12T09:04:43.489 INFO:tasks.workunit.client.0.smithi028.stdout: "minimum_required_version": "jewel", 2021-05-12T09:04:43.489 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables": 1, 2021-05-12T09:04:43.489 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables2": 1, 2021-05-12T09:04:43.489 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v2_rules": 0, 2021-05-12T09:04:43.490 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables3": 1, 2021-05-12T09:04:43.490 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v3_rules": 0, 2021-05-12T09:04:43.490 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v4_buckets": 1, 2021-05-12T09:04:43.490 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables5": 1, 2021-05-12T09:04:43.491 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v5_rules": 0 2021-05-12T09:04:43.491 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.491 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_args": {} 2021-05-12T09:04:43.491 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.492 INFO:tasks.workunit.client.0.smithi028.stdout: "fsmap": { 2021-05-12T09:04:43.492 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 1, 2021-05-12T09:04:43.492 INFO:tasks.workunit.client.0.smithi028.stdout: "default_fscid": -1, 2021-05-12T09:04:43.492 INFO:tasks.workunit.client.0.smithi028.stdout: "compat": { 2021-05-12T09:04:43.492 INFO:tasks.workunit.client.0.smithi028.stdout: "compat": {}, 2021-05-12T09:04:43.493 INFO:tasks.workunit.client.0.smithi028.stdout: "ro_compat": {}, 2021-05-12T09:04:43.493 INFO:tasks.workunit.client.0.smithi028.stdout: "incompat": { 2021-05-12T09:04:43.493 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_1": "base v0.20", 2021-05-12T09:04:43.493 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_2": "client writeable ranges", 2021-05-12T09:04:43.494 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T09:04:43.494 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_4": "dir inode in separate object", 2021-05-12T09:04:43.494 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T09:04:43.494 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T09:04:43.494 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_8": "no anchor table", 2021-05-12T09:04:43.495 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_9": "file layout v2", 2021-05-12T09:04:43.495 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_10": "snaprealm v2" 2021-05-12T09:04:43.495 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.495 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.496 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_flags": { 2021-05-12T09:04:43.496 INFO:tasks.workunit.client.0.smithi028.stdout: "enable_multiple": true, 2021-05-12T09:04:43.496 INFO:tasks.workunit.client.0.smithi028.stdout: "ever_enabled_multiple": true 2021-05-12T09:04:43.496 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.497 INFO:tasks.workunit.client.0.smithi028.stdout: "standbys": [], 2021-05-12T09:04:43.497 INFO:tasks.workunit.client.0.smithi028.stdout: "filesystems": [] 2021-05-12T09:04:43.497 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.497 INFO:tasks.workunit.client.0.smithi028.stdout: "mdsmap_first_committed": 1, 2021-05-12T09:04:43.497 INFO:tasks.workunit.client.0.smithi028.stdout: "mdsmap_last_committed": 1, 2021-05-12T09:04:43.498 INFO:tasks.workunit.client.0.smithi028.stdout: "auth": { 2021-05-12T09:04:43.498 INFO:tasks.workunit.client.0.smithi028.stdout: "first_committed": 1, 2021-05-12T09:04:43.498 INFO:tasks.workunit.client.0.smithi028.stdout: "last_committed": 5, 2021-05-12T09:04:43.498 INFO:tasks.workunit.client.0.smithi028.stdout: "num_secrets": 3 2021-05-12T09:04:43.499 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.499 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 2, 2021-05-12T09:04:43.499 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_active": 2, 2021-05-12T09:04:43.499 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_unknown": 0, 2021-05-12T09:04:43.500 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osd": 3, 2021-05-12T09:04:43.500 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_sum": { 2021-05-12T09:04:43.500 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:04:43.500 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 507580, 2021-05-12T09:04:43.500 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 102, 2021-05-12T09:04:43.501 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:04:43.501 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 306, 2021-05-12T09:04:43.501 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:04:43.501 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:04:43.502 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 102, 2021-05-12T09:04:43.502 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:04:43.502 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:04:43.502 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 102, 2021-05-12T09:04:43.503 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:04:43.503 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 76, 2021-05-12T09:04:43.503 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 64, 2021-05-12T09:04:43.503 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 195, 2021-05-12T09:04:43.503 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 1210, 2021-05-12T09:04:43.504 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:04:43.504 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:04:43.504 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:04:43.504 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:04:43.505 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:04:43.505 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:04:43.505 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:04:43.505 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:04:43.506 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:04:43.506 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:04:43.506 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:04:43.506 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:04:43.506 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:04:43.507 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:04:43.507 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:04:43.507 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:04:43.507 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:04:43.508 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:04:43.508 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:04:43.508 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:04:43.508 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:04:43.509 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:04:43.509 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:04:43.509 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:04:43.509 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:04:43.509 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.510 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:04:43.510 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:04:43.510 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:04:43.510 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:04:43.511 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 0, 2021-05-12T09:04:43.511 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 0, 2021-05-12T09:04:43.511 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:04:43.511 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:04:43.512 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:04:43.512 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:04:43.512 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:04:43.512 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.512 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 30, 2021-05-12T09:04:43.513 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 30, 2021-05-12T09:04:43.513 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 4, 2021-05-12T09:04:43.513 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 4, 2021-05-12T09:04:43.513 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 0 2021-05-12T09:04:43.514 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.514 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_sum": { 2021-05-12T09:04:43.514 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 0, 2021-05-12T09:04:43.514 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 0, 2021-05-12T09:04:43.515 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pgs": 6, 2021-05-12T09:04:43.515 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osds": 3, 2021-05-12T09:04:43.515 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_osds": 3, 2021-05-12T09:04:43.515 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T09:04:43.515 INFO:tasks.workunit.client.0.smithi028.stdout: "kb": 314572800, 2021-05-12T09:04:43.516 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used": 21252, 2021-05-12T09:04:43.516 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_data": 3560, 2021-05-12T09:04:43.516 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_omap": 2, 2021-05-12T09:04:43.516 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_meta": 17661, 2021-05-12T09:04:43.517 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_avail": 314551548, 2021-05-12T09:04:43.517 INFO:tasks.workunit.client.0.smithi028.stdout: "statfs": { 2021-05-12T09:04:43.517 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 322122547200, 2021-05-12T09:04:43.517 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 322100785152, 2021-05-12T09:04:43.518 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:04:43.518 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 3645440, 2021-05-12T09:04:43.518 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1854299, 2021-05-12T09:04:43.518 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:04:43.518 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:04:43.519 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:04:43.519 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 2160, 2021-05-12T09:04:43.519 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 18085776 2021-05-12T09:04:43.519 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.520 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_peers": [], 2021-05-12T09:04:43.520 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_trim_queue_len": 0, 2021-05-12T09:04:43.520 INFO:tasks.workunit.client.0.smithi028.stdout: "num_snap_trimming": 0, 2021-05-12T09:04:43.520 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shards_repaired": 1, 2021-05-12T09:04:43.521 INFO:tasks.workunit.client.0.smithi028.stdout: "op_queue_age_hist": { 2021-05-12T09:04:43.521 INFO:tasks.workunit.client.0.smithi028.stdout: "histogram": [], 2021-05-12T09:04:43.521 INFO:tasks.workunit.client.0.smithi028.stdout: "upper_bound": 1 2021-05-12T09:04:43.521 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.522 INFO:tasks.workunit.client.0.smithi028.stdout: "perf_stat": { 2021-05-12T09:04:43.522 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ms": 41, 2021-05-12T09:04:43.522 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ms": 41, 2021-05-12T09:04:43.522 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ns": 41000000, 2021-05-12T09:04:43.522 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ns": 41000000 2021-05-12T09:04:43.523 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.523 INFO:tasks.workunit.client.0.smithi028.stdout: "alerts": [], 2021-05-12T09:04:43.523 INFO:tasks.workunit.client.0.smithi028.stdout: "network_ping_times": [] 2021-05-12T09:04:43.523 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.524 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_sum_by_class": { 2021-05-12T09:04:43.524 INFO:tasks.workunit.client.0.smithi028.stdout: "hdd": { 2021-05-12T09:04:43.524 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 0, 2021-05-12T09:04:43.524 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 0, 2021-05-12T09:04:43.525 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pgs": 6, 2021-05-12T09:04:43.525 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osds": 3, 2021-05-12T09:04:43.525 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_osds": 3, 2021-05-12T09:04:43.525 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T09:04:43.525 INFO:tasks.workunit.client.0.smithi028.stdout: "kb": 314572800, 2021-05-12T09:04:43.526 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used": 21252, 2021-05-12T09:04:43.526 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_data": 3560, 2021-05-12T09:04:43.526 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_omap": 2, 2021-05-12T09:04:43.526 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_meta": 17661, 2021-05-12T09:04:43.527 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_avail": 314551548, 2021-05-12T09:04:43.527 INFO:tasks.workunit.client.0.smithi028.stdout: "statfs": { 2021-05-12T09:04:43.527 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 322122547200, 2021-05-12T09:04:43.527 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 322100785152, 2021-05-12T09:04:43.528 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:04:43.528 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 3645440, 2021-05-12T09:04:43.528 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1854299, 2021-05-12T09:04:43.528 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:04:43.528 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:04:43.529 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:04:43.529 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 2160, 2021-05-12T09:04:43.529 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 18085776 2021-05-12T09:04:43.529 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.530 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_peers": [], 2021-05-12T09:04:43.530 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_trim_queue_len": 0, 2021-05-12T09:04:43.530 INFO:tasks.workunit.client.0.smithi028.stdout: "num_snap_trimming": 0, 2021-05-12T09:04:43.530 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shards_repaired": 1, 2021-05-12T09:04:43.531 INFO:tasks.workunit.client.0.smithi028.stdout: "op_queue_age_hist": { 2021-05-12T09:04:43.531 INFO:tasks.workunit.client.0.smithi028.stdout: "histogram": [], 2021-05-12T09:04:43.531 INFO:tasks.workunit.client.0.smithi028.stdout: "upper_bound": 1 2021-05-12T09:04:43.531 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.531 INFO:tasks.workunit.client.0.smithi028.stdout: "perf_stat": { 2021-05-12T09:04:43.532 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ms": 41, 2021-05-12T09:04:43.532 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ms": 41, 2021-05-12T09:04:43.532 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ns": 41000000, 2021-05-12T09:04:43.532 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ns": 41000000 2021-05-12T09:04:43.533 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.533 INFO:tasks.workunit.client.0.smithi028.stdout: "alerts": [], 2021-05-12T09:04:43.533 INFO:tasks.workunit.client.0.smithi028.stdout: "network_ping_times": [] 2021-05-12T09:04:43.533 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.534 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.534 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_stats": [ 2021-05-12T09:04:43.534 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.534 INFO:tasks.workunit.client.0.smithi028.stdout: "poolid": 2, 2021-05-12T09:04:43.535 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 1, 2021-05-12T09:04:43.535 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:04:43.535 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 48300, 2021-05-12T09:04:43.535 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 100, 2021-05-12T09:04:43.535 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:04:43.536 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 300, 2021-05-12T09:04:43.536 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:04:43.536 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:04:43.536 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 100, 2021-05-12T09:04:43.537 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:04:43.537 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:04:43.537 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 100, 2021-05-12T09:04:43.537 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:04:43.537 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 0, 2021-05-12T09:04:43.538 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 0, 2021-05-12T09:04:43.538 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 100, 2021-05-12T09:04:43.538 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 100, 2021-05-12T09:04:43.538 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:04:43.539 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:04:43.539 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:04:43.539 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:04:43.539 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:04:43.540 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:04:43.540 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:04:43.540 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:04:43.540 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:04:43.540 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:04:43.541 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:04:43.541 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:04:43.541 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:04:43.541 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:04:43.542 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:04:43.542 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:04:43.542 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:04:43.542 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:04:43.543 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:04:43.543 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:04:43.543 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:04:43.543 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:04:43.543 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:04:43.544 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:04:43.544 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:04:43.544 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.544 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:04:43.545 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:04:43.545 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:04:43.545 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:04:43.545 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 1228800, 2021-05-12T09:04:43.546 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 144900, 2021-05-12T09:04:43.546 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:04:43.546 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:04:43.546 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:04:43.547 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:04:43.547 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:04:43.547 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.547 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 16, 2021-05-12T09:04:43.547 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 16, 2021-05-12T09:04:43.548 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 2, 2021-05-12T09:04:43.548 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 2, 2021-05-12T09:04:43.548 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 3 2021-05-12T09:04:43.548 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.549 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.549 INFO:tasks.workunit.client.0.smithi028.stdout: "poolid": 1, 2021-05-12T09:04:43.549 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 1, 2021-05-12T09:04:43.549 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:04:43.550 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 459280, 2021-05-12T09:04:43.550 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 2, 2021-05-12T09:04:43.550 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:04:43.550 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 6, 2021-05-12T09:04:43.551 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:04:43.551 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:04:43.551 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 2, 2021-05-12T09:04:43.551 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:04:43.551 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:04:43.552 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 2, 2021-05-12T09:04:43.552 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:04:43.552 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 76, 2021-05-12T09:04:43.552 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 64, 2021-05-12T09:04:43.553 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 95, 2021-05-12T09:04:43.553 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 1110, 2021-05-12T09:04:43.553 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:04:43.553 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:04:43.554 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:04:43.554 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:04:43.554 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:04:43.554 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:04:43.554 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:04:43.555 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:04:43.555 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:04:43.555 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:04:43.555 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:04:43.556 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:04:43.556 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:04:43.556 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:04:43.556 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:04:43.557 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:04:43.557 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:04:43.557 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:04:43.557 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:04:43.557 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:04:43.558 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:04:43.558 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:04:43.558 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:04:43.558 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:04:43.559 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:04:43.559 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.559 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:04:43.559 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:04:43.559 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:04:43.560 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:04:43.560 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 1388544, 2021-05-12T09:04:43.560 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1377840, 2021-05-12T09:04:43.560 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:04:43.561 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:04:43.561 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:04:43.561 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:04:43.561 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:04:43.562 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.562 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 14, 2021-05-12T09:04:43.562 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 14, 2021-05-12T09:04:43.562 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 2, 2021-05-12T09:04:43.563 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 2, 2021-05-12T09:04:43.563 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 3 2021-05-12T09:04:43.563 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.563 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.563 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_stats": [ 2021-05-12T09:04:43.564 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.564 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:04:43.564 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 163208757259 2021-05-12T09:04:43.564 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.565 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.565 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:04:43.565 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 171798691850 2021-05-12T09:04:43.565 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.565 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.566 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:04:43.566 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 64424509456 2021-05-12T09:04:43.566 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.566 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.567 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_by_state": [ 2021-05-12T09:04:43.567 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.567 INFO:tasks.workunit.client.0.smithi028.stdout: "state": "active+undersized+degraded", 2021-05-12T09:04:43.567 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 2 2021-05-12T09:04:43.568 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.568 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.568 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_by_osd": [ 2021-05-12T09:04:43.568 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.568 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:04:43.569 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:04:43.569 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 2, 2021-05-12T09:04:43.569 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:04:43.569 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.570 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.570 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:04:43.570 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:04:43.570 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 0, 2021-05-12T09:04:43.571 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:04:43.571 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.571 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.571 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:04:43.571 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 2, 2021-05-12T09:04:43.572 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 2, 2021-05-12T09:04:43.572 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:04:43.572 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.572 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.573 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": [ 2021-05-12T09:04:43.573 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.573 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 1, 2021-05-12T09:04:43.573 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": {} 2021-05-12T09:04:43.574 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.574 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:04:43.574 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 2, 2021-05-12T09:04:43.574 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": {} 2021-05-12T09:04:43.575 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.575 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:04:43.575 INFO:tasks.workunit.client.0.smithi028.stdout: "servicemap": { 2021-05-12T09:04:43.575 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 5, 2021-05-12T09:04:43.575 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:04:12.891875+0000", 2021-05-12T09:04:43.576 INFO:tasks.workunit.client.0.smithi028.stdout: "services": { 2021-05-12T09:04:43.576 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": { 2021-05-12T09:04:43.576 INFO:tasks.workunit.client.0.smithi028.stdout: "daemons": { 2021-05-12T09:04:43.576 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": "", 2021-05-12T09:04:43.577 INFO:tasks.workunit.client.0.smithi028.stdout: "0": { 2021-05-12T09:04:43.577 INFO:tasks.workunit.client.0.smithi028.stdout: "start_epoch": 0, 2021-05-12T09:04:43.577 INFO:tasks.workunit.client.0.smithi028.stdout: "start_stamp": "0.000000", 2021-05-12T09:04:43.577 INFO:tasks.workunit.client.0.smithi028.stdout: "gid": 0, 2021-05-12T09:04:43.577 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T09:04:43.578 INFO:tasks.workunit.client.0.smithi028.stdout: "metadata": {}, 2021-05-12T09:04:43.578 INFO:tasks.workunit.client.0.smithi028.stdout: "task_status": {} 2021-05-12T09:04:43.578 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.578 INFO:tasks.workunit.client.0.smithi028.stdout: "1": { 2021-05-12T09:04:43.579 INFO:tasks.workunit.client.0.smithi028.stdout: "start_epoch": 0, 2021-05-12T09:04:43.579 INFO:tasks.workunit.client.0.smithi028.stdout: "start_stamp": "0.000000", 2021-05-12T09:04:43.579 INFO:tasks.workunit.client.0.smithi028.stdout: "gid": 0, 2021-05-12T09:04:43.579 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T09:04:43.580 INFO:tasks.workunit.client.0.smithi028.stdout: "metadata": {}, 2021-05-12T09:04:43.580 INFO:tasks.workunit.client.0.smithi028.stdout: "task_status": {} 2021-05-12T09:04:43.580 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.580 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.580 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.581 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.581 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:04:43.581 INFO:tasks.workunit.client.0.smithi028.stdout: "paxos": { 2021-05-12T09:04:43.581 INFO:tasks.workunit.client.0.smithi028.stdout: "first_committed": 1, 2021-05-12T09:04:43.582 INFO:tasks.workunit.client.0.smithi028.stdout: "last_committed": 371, 2021-05-12T09:04:43.582 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pn": 0, 2021-05-12T09:04:43.582 INFO:tasks.workunit.client.0.smithi028.stdout: "accepted_pn": 0 2021-05-12T09:04:43.582 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:04:43.583 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:04:43.584 INFO:tasks.workunit.client.0.smithi028.stderr:report 3376381381 2021-05-12T09:04:43.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T09:04:43.584 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2021-05-12T09:04:43.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2021-05-12T09:04:43.585 INFO:tasks.workunit.client.0.smithi028.stdout:writing non-divergent object existing_4 2021-05-12T09:04:43.586 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2021-05-12T09:04:43.586 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2021-05-12T09:04:43.654 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:04:43.655 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T09:04:08.052277+0000 31'100 46:120 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:03:14.594281+0000 0'0 2021-05-12T09:03:14.594281+0000 0 2021-05-12T09:04:43.655 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:04:07.943459+0000 31'62 46:110 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:03:13.031503+0000 0'0 2021-05-12T09:03:13.031503+0000 0 2021-05-12T09:04:43.655 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:04:43.656 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:04:43.656 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:04:43.671 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2021-05-12T09:04:43.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2021-05-12T09:04:43.747 INFO:tasks.workunit.client.0.smithi028.stdout:delay recovery 2021-05-12T09:04:43.748 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2021-05-12T09:04:44.107 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:04:44.107 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T09:04:08.052277+0000 31'100 46:120 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:03:14.594281+0000 0'0 2021-05-12T09:03:14.594281+0000 0 2021-05-12T09:04:44.108 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:04:07.943459+0000 31'62 46:110 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:03:13.031503+0000 0'0 2021-05-12T09:03:13.031503+0000 0 2021-05-12T09:04:44.108 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:04:44.108 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:04:44.109 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:04:44.119 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2021-05-12T09:04:44.120 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2021-05-12T09:04:44.120 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T09:04:44.120 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T09:04:44.120 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:04:44.121 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:04:44.121 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:04:44.121 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.0.asok 2021-05-12T09:04:44.122 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2021-05-12T09:04:44.122 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.0.asok set_recovery_delay 100000 2021-05-12T09:04:44.220 INFO:tasks.workunit.client.0.smithi028.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T09:04:44.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2021-05-12T09:04:44.230 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.1 2021-05-12T09:04:44.230 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T09:04:44.230 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T09:04:44.230 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:04:44.231 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:04:44.231 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:04:44.231 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.1.asok 2021-05-12T09:04:44.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2021-05-12T09:04:44.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.1.asok set_recovery_delay 100000 2021-05-12T09:04:44.330 INFO:tasks.workunit.client.0.smithi028.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T09:04:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 2' 2021-05-12T09:04:44.339 INFO:tasks.workunit.client.0.smithi028.stdout:revive divergent 2 2021-05-12T09:04:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2021-05-12T09:04:44.700 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:04:44.701 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T09:04:08.052277+0000 31'100 46:120 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:03:14.594281+0000 0'0 2021-05-12T09:03:14.594281+0000 0 2021-05-12T09:04:44.701 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:04:07.943459+0000 31'62 46:110 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:03:13.031503+0000 0'0 2021-05-12T09:03:13.031503+0000 0 2021-05-12T09:04:44.701 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:04:44.702 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:04:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:04:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2021-05-12T09:04:45.349 INFO:tasks.workunit.client.0.smithi028.stderr:noup is set 2021-05-12T09:04:45.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 2 2021-05-12T09:04:45.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:04:45.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:04:45.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T09:04:45.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:04:45.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T09:04:45.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:04:45.364 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:04:45.364 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:04:45.364 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:04:45.364 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T09:04:45.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T09:04:45.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:04:45.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:04:45.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:04:45.365 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:04:45.366 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:04:45.366 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:04:45.366 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:04:45.366 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:04:45.367 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:04:45.367 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:04:45.367 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:04:45.367 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:04:45.367 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:04:45.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:04:45.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:04:45.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:04:45.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:04:45.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:04:45.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:04:45.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T09:04:45.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T09:04:45.370 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:04:45.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:04:45.371 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T09:04:45.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T09:04:45.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:04:45.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:04:45.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:04:45.391 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:04:45.391+0000 7fe3defc90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:04:45.392 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:04:45.393+0000 7fe3defc90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:04:45.394 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:04:45.394+0000 7fe3defc90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:04:45.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: return 0 2021-05-12T09:04:45.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2021-05-12T09:04:45.909 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:04:45.909+0000 7fe3defc90c0 -1 Falling back to public interface 2021-05-12T09:04:48.478 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:04:48.478+0000 7fe3defc90c0 -1 osd.2 31 log_to_monitors {default=true} 2021-05-12T09:04:50.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2021-05-12T09:04:50.752 INFO:tasks.workunit.client.0.smithi028.stdout:delay recovery divergent 2021-05-12T09:04:50.753 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2021-05-12T09:04:51.129 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:04:51.129 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T09:04:08.052277+0000 46'101 48:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:03:14.594281+0000 0'0 2021-05-12T09:03:14.594281+0000 0 2021-05-12T09:04:51.129 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:04:07.943459+0000 31'62 48:113 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:03:13.031503+0000 0'0 2021-05-12T09:03:13.031503+0000 0 2021-05-12T09:04:51.129 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:04:51.130 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:04:51.131 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:04:51.141 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.2 2021-05-12T09:04:51.141 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T09:04:51.141 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T09:04:51.142 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:04:51.142 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:04:51.142 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:04:51.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.2.asok 2021-05-12T09:04:51.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2021-05-12T09:04:51.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.2.asok set_recovery_delay 100000 2021-05-12T09:04:51.243 INFO:tasks.workunit.client.0.smithi028.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T09:04:51.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2021-05-12T09:04:52.143 INFO:tasks.workunit.client.0.smithi028.stderr:noup is unset 2021-05-12T09:04:52.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2021-05-12T09:04:52.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:04:52.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:04:52.155 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:04:52.155 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:04:52.155 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:04:52.155 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:04:52.156 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:04:52.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:04:52.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:04:52.541 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 38 up_thru 43 down_at 32 last_clean_interval [5,31) [v2:127.0.0.1:6802/37667,v1:127.0.0.1:6803/37667] [v2:127.0.0.1:6804/37667,v1:127.0.0.1:6805/37667] exists,up 76b38569-e8bc-43f9-a984-1867943d6e8d 2021-05-12T09:04:52.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:04:52.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:04:52.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:04:52.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2021-05-12T09:04:52.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:04:52.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:04:52.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:04:52.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:04:52.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:04:52.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:04:52.545 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:04:52.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:04:52.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:04:52.940 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 40 up_thru 51 down_at 33 last_clean_interval [10,31) [v2:127.0.0.1:6810/38232,v1:127.0.0.1:6811/38232] [v2:127.0.0.1:6812/38232,v1:127.0.0.1:6813/38232] exists,up fcbcbfab-6930-4442-a86e-6c4dd2cd2b5c 2021-05-12T09:04:52.941 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:04:52.941 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:04:52.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:04:52.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2021-05-12T09:04:52.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:04:52.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:04:52.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:04:52.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:04:52.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:04:52.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:04:52.944 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:04:52.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:04:52.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:04:53.348 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 51 up_thru 51 down_at 34 last_clean_interval [15,31) [v2:127.0.0.1:6818/39938,v1:127.0.0.1:6819/39938] [v2:127.0.0.1:6820/39938,v1:127.0.0.1:6821/39938] exists,up 786b0ee9-ebdd-46c7-9baf-4db98735ea54 2021-05-12T09:04:53.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:04:53.350 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:04:53.350 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:04:53.350 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2021-05-12T09:04:53.715 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:04:53.715 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T09:04:08.052277+0000 46'101 48:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:03:14.594281+0000 0'0 2021-05-12T09:03:14.594281+0000 0 2021-05-12T09:04:53.715 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:04:07.943459+0000 31'62 48:113 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:03:13.031503+0000 0'0 2021-05-12T09:03:13.031503+0000 0 2021-05-12T09:04:53.716 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:04:53.717 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:04:53.717 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:04:53.726 INFO:tasks.workunit.client.0.smithi028.stdout:wait for peering 2021-05-12T09:04:53.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2021-05-12T09:04:53.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2021-05-12T09:04:54.092 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:04:54.092 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T09:04:08.052277+0000 46'101 48:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:03:14.594281+0000 0'0 2021-05-12T09:03:14.594281+0000 0 2021-05-12T09:04:54.092 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:04:07.943459+0000 31'62 48:113 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:03:13.031503+0000 0'0 2021-05-12T09:03:13.031503+0000 0 2021-05-12T09:04:54.093 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:04:54.093 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:04:54.094 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:04:54.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2021-05-12T09:05:04.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 2' 2021-05-12T09:05:04.150 INFO:tasks.workunit.client.0.smithi028.stdout:killing divergent 2 2021-05-12T09:05:04.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2021-05-12T09:05:04.526 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:05:04.527 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2021-05-12T09:04:52.712627+0000 46'101 53:136 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:03:14.594281+0000 0'0 2021-05-12T09:03:14.594281+0000 0 2021-05-12T09:05:04.527 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:04:52.688832+0000 31'62 53:126 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:03:13.031503+0000 0'0 2021-05-12T09:03:13.031503+0000 0 2021-05-12T09:05:04.527 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:05:04.528 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:05:04.528 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:05:04.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.2 2021-05-12T09:05:04.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:05:04.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:05:04.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:05:04.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:05:04.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:05:04.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:05:04.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 2' 2021-05-12T09:05:04.644 INFO:tasks.workunit.client.0.smithi028.stdout:reviving divergent 2 2021-05-12T09:05:04.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2021-05-12T09:05:05.010 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:05:05.011 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2021-05-12T09:04:52.712627+0000 46'101 53:136 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:03:14.594281+0000 0'0 2021-05-12T09:03:14.594281+0000 0 2021-05-12T09:05:05.011 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:04:52.688832+0000 31'62 53:126 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:03:13.031503+0000 0'0 2021-05-12T09:03:13.031503+0000 0 2021-05-12T09:05:05.011 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:05:05.011 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:05:05.012 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:05:05.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 2 2021-05-12T09:05:05.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:05:05.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:05:05.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T09:05:05.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:05:05.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T09:05:05.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:05:05.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:05:05.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:05:05.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:05:05.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T09:05:05.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T09:05:05.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:05:05.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:05:05.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:05:05.028 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:05:05.029 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:05:05.029 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:05:05.029 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:05:05.029 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:05:05.029 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:05:05.030 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:05:05.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:05:05.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:05:05.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:05:05.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:05:05.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:05:05.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:05:05.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:05:05.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:05:05.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:05:05.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T09:05:05.032 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:05:05.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T09:05:05.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:05:05.034 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T09:05:05.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T09:05:05.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:05:05.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:05:05.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:05:05.053 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:05.053+0000 7ff7f80340c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:05.055 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:05.054+0000 7ff7f80340c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:05.056 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:05.055+0000 7ff7f80340c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:05.415 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T09:05:05.415 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:05:05.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:05:05.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:05:05.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:05:05.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:05:05.417 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:05:05.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:05:05.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:05:05.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:05:05.571 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:05.571+0000 7ff7f80340c0 -1 Falling back to public interface 2021-05-12T09:05:05.796 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 51 up_thru 51 down_at 34 last_clean_interval [15,31) [v2:127.0.0.1:6818/39938,v1:127.0.0.1:6819/39938] [v2:127.0.0.1:6820/39938,v1:127.0.0.1:6821/39938] exists,up 786b0ee9-ebdd-46c7-9baf-4db98735ea54 2021-05-12T09:05:05.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:05:05.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:05:05.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:05:05.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2021-05-12T09:05:07.994 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:07.993+0000 7ff7f80340c0 -1 osd.2 53 log_to_monitors {default=true} 2021-05-12T09:05:25.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2021-05-12T09:05:25.799 INFO:tasks.workunit.client.0.smithi028.stdout:allowing recovery 2021-05-12T09:05:25.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2021-05-12T09:05:26.173 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:05:26.173 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2021-05-12T09:05:12.811556+0000 53'102 58:154 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:03:14.594281+0000 0'0 2021-05-12T09:03:14.594281+0000 0 2021-05-12T09:05:26.173 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:05:09.412750+0000 31'62 58:140 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:03:13.031503+0000 0'0 2021-05-12T09:03:13.031503+0000 0 2021-05-12T09:05:26.174 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:05:26.174 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:05:26.175 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:05:26.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-05-12T09:05:26.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2021-05-12T09:05:26.315 INFO:tasks.workunit.client.0.smithi028.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T09:05:26.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-05-12T09:05:26.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2021-05-12T09:05:26.455 INFO:tasks.workunit.client.0.smithi028.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T09:05:26.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-05-12T09:05:26.464 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2021-05-12T09:05:26.594 INFO:tasks.workunit.client.0.smithi028.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T09:05:26.603 INFO:tasks.workunit.client.0.smithi028.stdout:reading divergent objects 2021-05-12T09:05:26.603 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2021-05-12T09:05:26.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2021-05-12T09:05:26.962 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:05:26.963 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2021-05-12T09:05:12.811556+0000 53'102 58:154 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:03:14.594281+0000 0'0 2021-05-12T09:03:14.594281+0000 0 2021-05-12T09:05:26.963 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:05:09.412750+0000 31'62 58:140 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:03:13.031503+0000 0'0 2021-05-12T09:03:13.031503+0000 0 2021-05-12T09:05:26.963 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:05:26.964 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:05:26.964 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:05:26.974 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2021-05-12T09:05:26.975 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2021-05-12T09:05:26.976 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:05:26.976 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2021-05-12T09:05:27.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:05:27.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2021-05-12T09:05:27.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:05:27.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2021-05-12T09:05:27.071 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:05:27.072 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2021-05-12T09:05:27.102 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2021-05-12T09:05:27.103 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2021-05-12T09:05:27.104 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T09:05:27.110 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:04:52.042+0000 7fe3baebe700 20 _merge_object_divergent_entries: merging hoid 2:c156033e:::existing_100:head entries: 31'101 (31'100) modify 2:c156033e:::existing_100:head by client.4534.0:1 2021-05-12T09:03:44.493110+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:05:27.110 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:04:52.042+0000 7fe3baebe700 20 _merge_object_divergent_entries: keeping 31'101 (31'100) modify 2:c156033e:::existing_100:head by client.4534.0:1 2021-05-12T09:03:44.493110+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:05:27.111 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:04:52.042+0000 7fe3baebe700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T09:05:27.111 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:04:52.042+0000 7fe3baebe700 10 _merge_object_divergent_entries: hoid 2:c156033e:::existing_100:head prior_version: 31'100 first_divergent_update: 31'101 last_divergent_update: 31'101 2021-05-12T09:05:27.111 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:04:52.042+0000 7fe3baebe700 10 _merge_object_divergent_entries: hoid 2:c156033e:::existing_100:head has no more recent entries in log 2021-05-12T09:05:27.111 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:04:52.042+0000 7fe3baebe700 10 _merge_object_divergent_entries: hoid 2:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2021-05-12T09:05:27.111 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:04:52.042+0000 7fe3baebe700 10 _merge_object_divergent_entries: hoid 2:c156033e:::existing_100:head olog_can_rollback_to: 31'101 2021-05-12T09:05:27.112 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:04:52.042+0000 7fe3baebe700 10 _merge_object_divergent_entries: hoid 2:c156033e:::existing_100:head cannot rollback 31'101 (31'100) modify 2:c156033e:::existing_100:head by client.4534.0:1 2021-05-12T09:03:44.493110+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:05:27.112 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:04:52.042+0000 7fe3baebe700 10 _merge_object_divergent_entries: hoid 2:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2021-05-12T09:05:27.113 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2021-05-12T09:05:27.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T09:05:27.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2021-05-12T09:05:27.116 INFO:tasks.workunit.client.0.smithi028.stdout:success 2021-05-12T09:05:27.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2021-05-12T09:05:27.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-05-12T09:05:27.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-05-12T09:05:27.598 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'test' does not exist 2021-05-12T09:05:27.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2021-05-12T09:05:27.608 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:05:27.608 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:05:27.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:05:27.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:05:27.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:05:27.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:05:27.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-05-12T09:05:27.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T09:05:27.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:05:27.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T09:05:27.717 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:05:27.718 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:05:27.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:05:27.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:05:27.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:05:27.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:05:27.721 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:05:27.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:05:27.722 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:05:27.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:05:27.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:05:27.723 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:05:27.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:05:27.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:05:27.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:05:27.726 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:05:27.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:05:27.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:05:27.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T09:05:27.739 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:05:27.739 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:05:27.740 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:05:27.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29719 2021-05-12T09:05:27.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:05:27.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:05:27.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-05-12T09:05:27.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-05-12T09:05:27.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-05-12T09:05:27.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-05-12T09:05:27.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T09:05:27.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:05:27.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T09:05:27.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:05:27.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:05:27.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:05:27.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:05:27.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:05:27.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:05:27.746 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:05:27.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:05:27.747 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:05:27.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:05:27.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:05:27.747 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:05:27.748 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:05:27.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:05:27.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:05:27.749 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:05:27.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:05:27.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:05:27.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T09:05:27.751 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:05:27.751 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:05:27.751 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:05:27.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29719 2021-05-12T09:05:27.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:05:27.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:05:27.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-05-12T09:05:27.754 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:05:27.755 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:05:27.755 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:05:27.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.29719 2021-05-12T09:05:27.755 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:05:27.756 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T09:05:27.756 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:05:27.756 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-05-12T09:05:27.756 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2021-05-12T09:05:27.757 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2021-05-12T09:05:27.757 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2021-05-12T09:05:27.757 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2021-05-12T09:05:27.757 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2021-05-12T09:05:27.758 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2021-05-12T09:05:27.759 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2021-05-12T09:05:27.759 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2021-05-12T09:05:27.760 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:05:27.760 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:05:27.760 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2021-05-12T09:05:27.760 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-05-12T09:05:27.760 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:05:27.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:05:27.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:05:27.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-05-12T09:05:27.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-05-12T09:05:28.114 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:05:28.114 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:05:28.114 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:05:28.115 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:05:28.115 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:05:28.115 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:05:28.115 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:05:28.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:05:28.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:05:28.383 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:05:28.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:05:28.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:05:28.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:05:28.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:05:28.384 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:05:28.385 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:05:28.385 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:05:28.385 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:05:28.385 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:05:28.386 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:05:28.386 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-mon.a.asok 2021-05-12T09:05:28.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:05:28.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29719/ceph-mon.a.asok config get fsid 2021-05-12T09:05:28.484 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:05:28.485 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:05:28.485 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:05:28.485 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:05:28.486 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:05:28.486 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:05:28.486 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:05:28.486 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:05:28.487 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:05:28.487 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:05:28.487 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:05:28.487 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-mon.a.asok 2021-05-12T09:05:28.488 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:05:28.488 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29719/ceph-mon.a.asok config get mon_host 2021-05-12T09:05:28.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2021-05-12T09:05:28.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-05-12T09:05:28.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:05:28.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:05:28.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:05:28.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-05-12T09:05:28.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:05:29.071 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:05:29.071 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:05:29.071 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:05:29.072 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:05:29.072 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:05:29.072 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:05:29.072 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:05:29.073 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:05:29.073 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:05:29.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-05-12T09:05:29.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2021-05-12T09:05:29.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T09:05:29.104 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:05:29.104 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:05:29.104 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:05:29.104 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-05-12T09:05:29.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:05:29.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:05:29.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:05:29.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:05:29.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T09:05:29.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T09:05:29.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:05:29.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:05:29.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:05:29.107 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:05:29.107 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:05:29.107 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:05:29.108 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:05:29.108 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:05:29.108 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:05:29.108 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:05:29.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:05:29.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:05:29.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:05:29.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:05:29.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:05:29.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:05:29.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:05:29.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:05:29.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:05:29.111 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:05:29.111 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:05:29.111 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/0 2021-05-12T09:05:29.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:05:29.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=c5a51d7a-897f-41a2-97c3-e9ef91d860e1 2021-05-12T09:05:29.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 c5a51d7a-897f-41a2-97c3-e9ef91d860e1' 2021-05-12T09:05:29.113 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 c5a51d7a-897f-41a2-97c3-e9ef91d860e1 2021-05-12T09:05:29.113 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:05:29.123 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBZmptgl29eBxAAtoE4C3+jugLOE9yiPWOBUQ== 2021-05-12T09:05:29.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBZmptgl29eBxAAtoE4C3+jugLOE9yiPWOBUQ=="}' 2021-05-12T09:05:29.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new c5a51d7a-897f-41a2-97c3-e9ef91d860e1 -i td/divergent-priors/0/new.json 2021-05-12T09:05:29.449 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:05:29.460 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/0/new.json 2021-05-12T09:05:29.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBZmptgl29eBxAAtoE4C3+jugLOE9yiPWOBUQ== --osd-uuid c5a51d7a-897f-41a2-97c3-e9ef91d860e1 2021-05-12T09:05:29.492 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:29.491+0000 7f185e29a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:29.494 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:29.493+0000 7f185e29a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:29.495 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:29.494+0000 7f185e29a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:29.495 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:29.494+0000 7f185e29a0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T09:05:29.496 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:29.494+0000 7f185e29a0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T09:05:29.496 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:29.494+0000 7f185e29a0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T09:05:29.556 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:29.555+0000 7f185e29a0c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-05-12T09:05:33.848 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-05-12T09:05:33.848 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:05:33.849 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:05:33.849 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:05:33.850 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:05:34.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:05:34.107 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:05:34.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:05:34.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:05:34.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:05:34.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:05:34.136 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:34.135+0000 7f3e2a22a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:34.137 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:34.137+0000 7f3e2a22a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:34.139 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:34.138+0000 7f3e2a22a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:34.493 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:05:34.493 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:05:34.493 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:05:34.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:05:34.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:05:34.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:05:34.495 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:05:34.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:05:34.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:05:34.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:05:34.654 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:34.653+0000 7f3e2a22a0c0 -1 Falling back to public interface 2021-05-12T09:05:34.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:05:35.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:05:35.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:05:35.883 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:05:35.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:05:35.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:05:35.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:05:36.298 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:05:36.826 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:36.825+0000 7f3e2a22a0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:05:37.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:05:37.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:05:37.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:05:37.300 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:05:37.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:05:37.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:05:37.705 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/42571,v1:127.0.0.1:6803/42571] [v2:127.0.0.1:6804/42571,v1:127.0.0.1:6805/42571] exists,up c5a51d7a-897f-41a2-97c3-e9ef91d860e1 2021-05-12T09:05:37.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:05:37.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:05:37.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:05:37.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-05-12T09:05:37.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2021-05-12T09:05:37.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T09:05:37.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:05:37.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:05:37.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:05:37.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-05-12T09:05:37.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:05:37.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:05:37.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:05:37.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:05:37.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T09:05:37.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T09:05:37.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:05:37.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:05:37.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:05:37.714 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:05:37.715 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:05:37.715 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:05:37.715 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:05:37.715 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:05:37.716 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:05:37.717 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:05:37.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:05:37.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:05:37.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:05:37.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:05:37.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:05:37.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:05:37.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:05:37.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:05:37.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:05:37.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:05:37.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:05:37.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/1 2021-05-12T09:05:37.720 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:05:37.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=e003828b-f9e6-48de-bd5a-128e4af3e5dc 2021-05-12T09:05:37.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 e003828b-f9e6-48de-bd5a-128e4af3e5dc' 2021-05-12T09:05:37.721 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 e003828b-f9e6-48de-bd5a-128e4af3e5dc 2021-05-12T09:05:37.721 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:05:37.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBhmptgn91oKxAAxFqedoOTdKdat+YmmKZ7sw== 2021-05-12T09:05:37.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBhmptgn91oKxAAxFqedoOTdKdat+YmmKZ7sw=="}' 2021-05-12T09:05:37.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new e003828b-f9e6-48de-bd5a-128e4af3e5dc -i td/divergent-priors/1/new.json 2021-05-12T09:05:38.208 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:05:38.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/1/new.json 2021-05-12T09:05:38.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBhmptgn91oKxAAxFqedoOTdKdat+YmmKZ7sw== --osd-uuid e003828b-f9e6-48de-bd5a-128e4af3e5dc 2021-05-12T09:05:38.252 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:38.251+0000 7ff6346360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:38.253 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:38.252+0000 7ff6346360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:38.254 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:38.253+0000 7ff6346360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:38.255 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:38.254+0000 7ff6346360c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T09:05:38.255 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:38.254+0000 7ff6346360c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T09:05:38.255 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:38.254+0000 7ff6346360c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T09:05:38.320 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:38.319+0000 7ff6346360c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-05-12T09:05:42.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-05-12T09:05:42.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:05:42.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:05:42.582 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:05:42.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:05:43.104 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:05:43.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:05:43.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:05:43.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:05:43.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:05:43.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:05:43.134 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:43.133+0000 7f8b0794e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:43.136 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:43.135+0000 7f8b0794e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:43.137 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:43.136+0000 7f8b0794e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:43.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:05:43.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:05:43.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:05:43.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:05:43.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:05:43.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:05:43.502 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:05:43.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:05:43.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:05:43.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:05:43.654 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:43.653+0000 7f8b0794e0c0 -1 Falling back to public interface 2021-05-12T09:05:43.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:05:44.891 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:05:44.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:05:44.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:05:44.892 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:05:44.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:05:44.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:05:45.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:05:45.840 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:45.839+0000 7f8b0794e0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:05:46.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:05:46.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:05:46.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:05:46.291 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:05:46.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:05:46.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:05:46.691 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/43956,v1:127.0.0.1:6811/43956] [v2:127.0.0.1:6812/43956,v1:127.0.0.1:6813/43956] exists,up e003828b-f9e6-48de-bd5a-128e4af3e5dc 2021-05-12T09:05:46.697 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:05:46.697 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:05:46.697 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:05:46.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-05-12T09:05:46.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2021-05-12T09:05:46.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T09:05:46.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:05:46.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T09:05:46.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:05:46.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-05-12T09:05:46.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:05:46.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:05:46.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:05:46.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:05:46.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T09:05:46.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T09:05:46.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:05:46.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:05:46.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:05:46.702 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:05:46.702 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:05:46.703 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:05:46.703 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:05:46.703 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:05:46.703 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:05:46.704 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:05:46.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:05:46.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:05:46.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:05:46.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:05:46.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:05:46.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:05:46.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:05:46.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:05:46.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:05:46.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:05:46.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:05:46.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/2 2021-05-12T09:05:46.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:05:46.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=10e9d857-0855-44ba-8b8e-8d823ab9204a 2021-05-12T09:05:46.707 INFO:tasks.workunit.client.0.smithi028.stdout:add osd2 10e9d857-0855-44ba-8b8e-8d823ab9204a 2021-05-12T09:05:46.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 10e9d857-0855-44ba-8b8e-8d823ab9204a' 2021-05-12T09:05:46.708 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:05:46.715 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBqmptgu9SfKhAAZnN9ddMsSmgQt9CCCrQ+1Q== 2021-05-12T09:05:46.715 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBqmptgu9SfKhAAZnN9ddMsSmgQt9CCCrQ+1Q=="}' 2021-05-12T09:05:46.715 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 10e9d857-0855-44ba-8b8e-8d823ab9204a -i td/divergent-priors/2/new.json 2021-05-12T09:05:47.230 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:05:47.241 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/2/new.json 2021-05-12T09:05:47.242 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBqmptgu9SfKhAAZnN9ddMsSmgQt9CCCrQ+1Q== --osd-uuid 10e9d857-0855-44ba-8b8e-8d823ab9204a 2021-05-12T09:05:47.273 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:47.273+0000 7fbabb6d60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:47.275 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:47.275+0000 7fbabb6d60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:47.278 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:47.276+0000 7fbabb6d60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:47.279 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:47.277+0000 7fbabb6d60c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T09:05:47.279 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:47.277+0000 7fbabb6d60c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T09:05:47.279 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:47.277+0000 7fbabb6d60c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T09:05:47.319 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:47.319+0000 7fbabb6d60c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-05-12T09:05:51.887 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-05-12T09:05:51.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:05:51.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T09:05:51.889 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd2 key to auth repository 2021-05-12T09:05:51.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:05:52.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T09:05:52.419 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:05:52.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:05:52.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:05:52.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:05:52.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:05:52.449 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:52.449+0000 7f1ec6d650c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:52.450 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:52.450+0000 7f1ec6d650c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:52.451 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:52.452+0000 7f1ec6d650c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:05:52.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T09:05:52.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:05:52.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:05:52.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:05:52.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:05:52.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:05:52.811 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:05:52.811 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:05:52.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:05:52.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:05:52.973 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:52.973+0000 7f1ec6d650c0 -1 Falling back to public interface 2021-05-12T09:05:53.210 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:05:54.211 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:05:54.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:05:54.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:05:54.213 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:05:54.213 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:05:54.213 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:05:54.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:05:55.155 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:05:55.155+0000 7f1ec6d650c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T09:05:55.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:05:55.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:05:55.609 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:05:55.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:05:55.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:05:55.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:05:56.025 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/45341,v1:127.0.0.1:6819/45341] [v2:127.0.0.1:6820/45341,v1:127.0.0.1:6821/45341] exists,up 10e9d857-0855-44ba-8b8e-8d823ab9204a 2021-05-12T09:05:56.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:05:56.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:05:56.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:05:56.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2021-05-12T09:05:57.150 INFO:tasks.workunit.client.0.smithi028.stderr:noout is set 2021-05-12T09:05:57.160 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2021-05-12T09:05:58.106 INFO:tasks.workunit.client.0.smithi028.stderr:noin is set 2021-05-12T09:05:58.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2021-05-12T09:05:58.969 INFO:tasks.workunit.client.0.smithi028.stderr:nodown is set 2021-05-12T09:05:58.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2021-05-12T09:05:58.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-05-12T09:05:59.593 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'test' already exists 2021-05-12T09:05:59.603 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:06:00.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2021-05-12T09:06:01.625 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 2 size to 3 2021-05-12T09:06:01.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2021-05-12T09:06:02.600 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 2 min_size to 2 2021-05-12T09:06:02.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2021-05-12T09:06:02.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:06:02.610 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:06:03.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:06:03.012 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:06:03.012 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:06:03.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:06:03.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:06:03.014 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:06:03.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836487 2021-05-12T09:06:03.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836487 2021-05-12T09:06:03.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487' 2021-05-12T09:06:03.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:06:03.152 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:06:03.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672966 2021-05-12T09:06:03.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672966 2021-05-12T09:06:03.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-42949672966' 2021-05-12T09:06:03.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:06:03.294 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:06:03.434 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509444 2021-05-12T09:06:03.434 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509444 2021-05-12T09:06:03.434 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-42949672966 2-64424509444' 2021-05-12T09:06:03.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:06:03.436 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-05-12T09:06:03.436 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:06:03.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:06:03.436 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836487 2021-05-12T09:06:03.437 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:06:03.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836487 2021-05-12T09:06:03.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-05-12T09:06:03.438 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836487 2021-05-12T09:06:03.439 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:06:03.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836485 -lt 21474836487 2021-05-12T09:06:03.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:06:04.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:06:04.831 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:06:05.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836487 2021-05-12T09:06:05.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:06:05.227 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672966 2021-05-12T09:06:05.227 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:06:05.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:06:05.235 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672966 2021-05-12T09:06:05.236 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:06:05.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672966 2021-05-12T09:06:05.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-05-12T09:06:05.237 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672966 2021-05-12T09:06:05.237 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:06:05.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672966 -lt 42949672966 2021-05-12T09:06:05.731 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:06:05.731 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-05-12T09:06:05.732 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:06:05.734 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:06:05.736 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509444 2021-05-12T09:06:05.736 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:06:05.737 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509444 2021-05-12T09:06:05.737 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-05-12T09:06:05.739 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509444 2021-05-12T09:06:05.740 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:06:06.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509444 -lt 64424509444 2021-05-12T09:06:06.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2021-05-12T09:06:06.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:06:06.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:06:06.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:06:06.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:06:06.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T09:06:06.062 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:06:06.064 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:06:06.064 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:06:06.065 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:06:06.065 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:06:06.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:06:06.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:06:06.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:06:06.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:06:06.154 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:06:06.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:06:06.539 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:06:06.540 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:06:06.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:06:06.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:06:06.541 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:06:06.689 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836490 2021-05-12T09:06:06.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836490 2021-05-12T09:06:06.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490' 2021-05-12T09:06:06.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:06:06.690 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:06:06.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672968 2021-05-12T09:06:06.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672968 2021-05-12T09:06:06.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2021-05-12T09:06:06.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:06:06.830 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:06:06.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509447 2021-05-12T09:06:06.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509447 2021-05-12T09:06:06.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509447' 2021-05-12T09:06:06.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:06:06.971 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-05-12T09:06:06.971 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:06:06.972 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:06:06.972 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836490 2021-05-12T09:06:06.972 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:06:06.973 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836490 2021-05-12T09:06:06.973 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-05-12T09:06:06.973 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836490 2021-05-12T09:06:06.974 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:06:07.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836490 2021-05-12T09:06:07.358 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:06:08.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:06:08.359 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:06:08.759 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836490 -lt 21474836490 2021-05-12T09:06:08.760 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:06:08.763 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2021-05-12T09:06:08.764 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:06:08.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:06:08.764 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672968 2021-05-12T09:06:08.764 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:06:08.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672968 2021-05-12T09:06:08.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-05-12T09:06:08.765 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672968 2021-05-12T09:06:08.765 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:06:09.160 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672969 -lt 42949672968 2021-05-12T09:06:09.161 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:06:09.161 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509447 2021-05-12T09:06:09.162 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:06:09.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:06:09.163 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509447 2021-05-12T09:06:09.163 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:06:09.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509447 2021-05-12T09:06:09.164 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2021-05-12T09:06:09.164 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509447 2021-05-12T09:06:09.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:06:09.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509447 -lt 64424509447 2021-05-12T09:06:09.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:06:09.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:06:09.553 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:06:10.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:06:10.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:06:10.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:06:10.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:06:10.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:06:10.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:06:10.063 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:06:10.063 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:06:10.434 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T09:06:10.435 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:06:10.435 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:06:10.436 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:06:10.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T09:06:10.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:06:10.939 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:06:10.939 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2021-05-12T09:06:10.939 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2021-05-12T09:06:11.303 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:06:11.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=2 2021-05-12T09:06:11.313 INFO:tasks.workunit.client.0.smithi028.stdout:primary and soon to be divergent is 2 2021-05-12T09:06:11.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 2' 2021-05-12T09:06:11.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2021-05-12T09:06:11.676 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:06:11.677 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T09:06:00.517181+0000 0'0 31:14 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:05:59.322736+0000 0'0 2021-05-12T09:05:59.322736+0000 0 2021-05-12T09:06:11.677 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:05:56.863329+0000 32'62 32:107 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:05:55.908046+0000 0'0 2021-05-12T09:05:55.908046+0000 0 2021-05-12T09:06:11.677 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:06:11.678 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:06:11.678 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:06:11.688 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2021-05-12T09:06:11.689 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-05-12T09:06:11.689 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 2 ']' 2021-05-12T09:06:11.689 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2021-05-12T09:06:11.689 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-05-12T09:06:11.689 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 2 ']' 2021-05-12T09:06:11.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 1' 2021-05-12T09:06:11.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-05-12T09:06:11.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 2 ']' 2021-05-12T09:06:11.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2021-05-12T09:06:11.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2021-05-12T09:06:11.691 INFO:tasks.workunit.client.0.smithi028.stdout:writing initial objects 2021-05-12T09:06:11.692 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2021-05-12T09:06:11.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:11.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2021-05-12T09:06:11.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:11.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2021-05-12T09:06:11.939 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:11.939 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2021-05-12T09:06:12.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:12.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2021-05-12T09:06:12.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:12.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2021-05-12T09:06:12.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:12.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2021-05-12T09:06:12.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:12.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2021-05-12T09:06:12.550 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:12.550 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2021-05-12T09:06:12.659 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:12.659 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2021-05-12T09:06:12.776 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:12.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2021-05-12T09:06:12.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:12.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2021-05-12T09:06:13.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:13.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2021-05-12T09:06:13.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:13.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2021-05-12T09:06:13.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:13.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2021-05-12T09:06:13.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:13.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2021-05-12T09:06:13.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:13.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2021-05-12T09:06:13.614 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:13.614 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2021-05-12T09:06:13.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:13.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2021-05-12T09:06:13.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:13.891 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2021-05-12T09:06:13.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:14.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2021-05-12T09:06:14.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:14.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2021-05-12T09:06:14.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:14.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2021-05-12T09:06:14.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:14.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2021-05-12T09:06:14.460 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:14.460 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2021-05-12T09:06:14.627 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:14.628 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2021-05-12T09:06:14.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:14.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2021-05-12T09:06:14.853 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:14.854 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2021-05-12T09:06:14.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:14.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2021-05-12T09:06:15.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:15.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2021-05-12T09:06:15.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:15.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2021-05-12T09:06:15.331 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:15.331 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2021-05-12T09:06:15.448 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:15.449 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2021-05-12T09:06:15.557 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:15.558 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2021-05-12T09:06:15.666 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:15.667 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2021-05-12T09:06:15.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:15.776 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2021-05-12T09:06:15.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:15.955 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2021-05-12T09:06:16.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:16.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2021-05-12T09:06:16.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:16.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2021-05-12T09:06:16.370 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:16.370 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2021-05-12T09:06:16.520 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:16.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2021-05-12T09:06:16.629 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:16.629 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2021-05-12T09:06:16.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:16.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2021-05-12T09:06:16.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:16.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2021-05-12T09:06:16.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:16.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2021-05-12T09:06:17.123 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:17.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2021-05-12T09:06:17.241 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:17.242 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2021-05-12T09:06:17.358 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:17.358 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2021-05-12T09:06:17.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:17.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2021-05-12T09:06:17.626 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:17.627 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2021-05-12T09:06:17.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:17.753 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2021-05-12T09:06:17.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:17.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2021-05-12T09:06:18.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:18.029 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2021-05-12T09:06:18.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:18.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2021-05-12T09:06:18.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:18.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2021-05-12T09:06:18.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:18.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2021-05-12T09:06:18.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:18.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2021-05-12T09:06:18.633 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:18.633 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2021-05-12T09:06:18.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:18.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2021-05-12T09:06:18.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:18.908 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2021-05-12T09:06:19.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:19.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2021-05-12T09:06:19.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:19.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2021-05-12T09:06:19.268 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:19.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2021-05-12T09:06:19.394 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:19.394 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2021-05-12T09:06:19.569 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:19.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2021-05-12T09:06:19.686 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:19.687 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2021-05-12T09:06:19.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:19.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2021-05-12T09:06:19.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:19.930 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2021-05-12T09:06:20.055 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:20.056 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2021-05-12T09:06:20.164 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:20.164 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2021-05-12T09:06:20.298 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:20.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2021-05-12T09:06:20.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:20.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2021-05-12T09:06:20.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:20.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2021-05-12T09:06:20.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:20.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2021-05-12T09:06:20.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:20.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2021-05-12T09:06:20.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:20.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2021-05-12T09:06:21.069 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:21.069 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2021-05-12T09:06:21.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:21.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2021-05-12T09:06:21.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:21.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2021-05-12T09:06:21.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:21.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2021-05-12T09:06:21.588 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:21.588 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2021-05-12T09:06:21.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:21.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2021-05-12T09:06:21.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:21.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2021-05-12T09:06:21.964 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:21.965 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2021-05-12T09:06:22.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:22.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2021-05-12T09:06:22.258 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:22.258 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2021-05-12T09:06:22.367 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:22.367 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2021-05-12T09:06:22.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:22.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2021-05-12T09:06:22.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:22.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2021-05-12T09:06:22.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:22.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2021-05-12T09:06:22.894 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:22.894 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2021-05-12T09:06:23.019 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:23.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2021-05-12T09:06:23.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:23.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2021-05-12T09:06:23.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:23.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2021-05-12T09:06:23.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:23.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2021-05-12T09:06:23.505 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:23.506 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2021-05-12T09:06:23.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:23.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2021-05-12T09:06:23.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:23.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2021-05-12T09:06:23.933 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:23.933 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2021-05-12T09:06:24.075 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:24.075 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2021-05-12T09:06:24.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-05-12T09:06:24.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2021-05-12T09:06:24.319 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T09:06:24.319 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2021-05-12T09:06:24.319 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:06:24.320 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:06:24.320 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:06:24.320 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:06:24.320 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T09:06:24.320 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:06:24.321 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:06:24.321 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:06:24.321 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:06:24.321 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:06:24.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:06:24.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:06:24.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:06:24.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:06:24.381 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:06:24.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:06:24.765 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:06:24.765 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:06:24.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:06:24.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:06:24.765 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:06:24.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836496 2021-05-12T09:06:24.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836496 2021-05-12T09:06:24.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496' 2021-05-12T09:06:24.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:06:24.907 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:06:25.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672974 2021-05-12T09:06:25.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672974 2021-05-12T09:06:25.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496 1-42949672974' 2021-05-12T09:06:25.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:06:25.048 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:06:25.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509452 2021-05-12T09:06:25.187 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509452 2021-05-12T09:06:25.187 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509452' 2021-05-12T09:06:25.187 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:06:25.188 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2021-05-12T09:06:25.188 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:06:25.189 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:06:25.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836496 2021-05-12T09:06:25.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:06:25.190 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836496 2021-05-12T09:06:25.190 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-05-12T09:06:25.191 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836496 2021-05-12T09:06:25.192 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:06:25.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836494 -lt 21474836496 2021-05-12T09:06:25.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:06:26.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:06:26.575 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:06:26.973 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836496 -lt 21474836496 2021-05-12T09:06:26.974 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:06:26.974 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-05-12T09:06:26.974 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:06:26.975 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:06:26.975 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672974 2021-05-12T09:06:26.975 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:06:26.976 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672974 2021-05-12T09:06:26.976 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-05-12T09:06:26.976 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672974 2021-05-12T09:06:26.977 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:06:27.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672974 -lt 42949672974 2021-05-12T09:06:27.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:06:27.363 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2021-05-12T09:06:27.363 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:06:27.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:06:27.366 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509452 2021-05-12T09:06:27.366 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:06:27.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509452 2021-05-12T09:06:27.367 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-05-12T09:06:27.368 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509452 2021-05-12T09:06:27.368 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:06:27.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509452 -lt 64424509452 2021-05-12T09:06:27.749 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:06:27.750 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:06:27.750 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:06:28.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:06:28.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:06:28.264 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:06:28.264 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:06:28.264 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:06:28.264 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:06:28.265 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:06:28.265 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:06:28.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T09:06:28.645 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:06:28.645 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:06:28.645 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:06:29.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T09:06:29.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:06:29.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:06:29.153 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2021-05-12T09:06:29.153 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local poolname=test 2021-05-12T09:06:29.153 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: local objectname=existing_1 2021-05-12T09:06:29.154 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: ceph --format json osd map test existing_1 2021-05-12T09:06:29.154 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: jq -r .pgid 2021-05-12T09:06:29.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=2.0 2021-05-12T09:06:29.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 1' 2021-05-12T09:06:29.544 INFO:tasks.workunit.client.0.smithi028.stdout:blackholing osds 0 1 2021-05-12T09:06:29.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2021-05-12T09:06:29.919 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:06:29.920 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-05-12T09:06:00.517181+0000 32'100 32:114 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:05:59.322736+0000 0'0 2021-05-12T09:05:59.322736+0000 0 2021-05-12T09:06:29.920 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:05:56.863329+0000 32'62 32:107 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:05:55.908046+0000 0'0 2021-05-12T09:05:55.908046+0000 0 2021-05-12T09:06:29.920 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:06:29.921 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:06:29.921 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:06:29.932 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2021-05-12T09:06:29.932 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2021-05-12T09:06:29.933 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T09:06:29.933 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T09:06:29.933 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:06:29.933 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:06:29.933 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:06:29.934 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.0.asok 2021-05-12T09:06:29.934 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2021-05-12T09:06:29.934 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.0.asok config set objectstore_blackhole 1 2021-05-12T09:06:30.029 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:06:30.029 INFO:tasks.workunit.client.0.smithi028.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T09:06:30.029 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:06:30.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2021-05-12T09:06:30.038 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.1 2021-05-12T09:06:30.038 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T09:06:30.038 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T09:06:30.038 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:06:30.039 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:06:30.039 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:06:30.039 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.1.asok 2021-05-12T09:06:30.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2021-05-12T09:06:30.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.1.asok config set objectstore_blackhole 1 2021-05-12T09:06:30.133 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:06:30.133 INFO:tasks.workunit.client.0.smithi028.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T09:06:30.134 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:06:30.141 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2021-05-12T09:06:30.142 INFO:tasks.workunit.client.0.smithi028.stdout:create new divergent objects 2021-05-12T09:06:30.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2021-05-12T09:06:30.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T09:06:30.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T09:06:30.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2021-05-12T09:06:30.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2021-05-12T09:06:30.144 INFO:tasks.workunit.client.0.smithi028.stdout:writing divergent objects 2021-05-12T09:06:30.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2021-05-12T09:06:30.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2021-05-12T09:06:30.146 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2021-05-12T09:06:30.146 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2021-05-12T09:06:30.146 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2021-05-12T09:06:30.146 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2021-05-12T09:06:30.147 INFO:tasks.workunit.client.0.smithi028.stdout:remove divergent objects 2021-05-12T09:06:30.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2021-05-12T09:06:30.148 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2021-05-12T09:06:30.149 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-05-12T09:06:30.149 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2021-05-12T09:06:30.149 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2021-05-12T09:06:30.149 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-05-12T09:06:30.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2021-05-12T09:06:30.150 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2021-05-12T09:06:30.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2021-05-12T09:06:30.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2021-05-12T09:06:30.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2021-05-12T09:06:40.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2021-05-12T09:06:40.152 INFO:tasks.workunit.client.0.smithi028.stderr:rados: no process found 2021-05-12T09:06:40.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2021-05-12T09:06:40.153 INFO:tasks.workunit.client.0.smithi028.stdout:killing all the osds 2021-05-12T09:06:40.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2021-05-12T09:06:40.528 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:06:40.528 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2021-05-12T09:06:00.517181+0000 32'106 32:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:05:59.322736+0000 0'0 2021-05-12T09:05:59.322736+0000 0 2021-05-12T09:06:40.529 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:05:56.863329+0000 32'62 32:107 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:05:55.908046+0000 0'0 2021-05-12T09:05:55.908046+0000 0 2021-05-12T09:06:40.529 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:06:40.529 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:06:40.530 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:06:40.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2021-05-12T09:06:40.540 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:06:40.540 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:06:40.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:06:40.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:06:40.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:06:40.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:06:40.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-05-12T09:06:40.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2021-05-12T09:06:41.168 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already down. 2021-05-12T09:06:41.178 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-05-12T09:06:41.178 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2021-05-12T09:06:41.699 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already down. 2021-05-12T09:06:41.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-05-12T09:06:41.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2021-05-12T09:06:42.260 INFO:tasks.workunit.client.0.smithi028.stderr:osd.2 is already down. 2021-05-12T09:06:42.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2021-05-12T09:06:42.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2021-05-12T09:06:42.796 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already out. 2021-05-12T09:06:42.806 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2021-05-12T09:06:42.806 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.1 2021-05-12T09:06:43.335 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already out. 2021-05-12T09:06:43.344 INFO:tasks.workunit.client.0.smithi028.stdout:bringing up non_divergent 0 1 2021-05-12T09:06:43.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 1' 2021-05-12T09:06:43.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2021-05-12T09:06:43.708 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:06:43.708 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2021-05-12T09:06:00.517181+0000 32'106 32:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:05:59.322736+0000 0'0 2021-05-12T09:05:59.322736+0000 0 2021-05-12T09:06:43.708 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 stale+active+clean 2021-05-12T09:05:56.863329+0000 32'62 32:107 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:05:55.908046+0000 0'0 2021-05-12T09:05:55.908046+0000 0 2021-05-12T09:06:43.709 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:06:43.709 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:06:43.710 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:06:43.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2021-05-12T09:06:43.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2021-05-12T09:06:43.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:06:43.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:06:43.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T09:06:43.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:06:43.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T09:06:43.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:06:43.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:06:43.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:06:43.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:06:43.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T09:06:43.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T09:06:43.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:06:43.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:06:43.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:06:43.726 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:06:43.726 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:06:43.727 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:06:43.727 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:06:43.727 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:06:43.727 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:06:43.727 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:06:43.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:06:43.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:06:43.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:06:43.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:06:43.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:06:43.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:06:43.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:06:43.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:06:43.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:06:43.730 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T09:06:43.730 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:06:43.731 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T09:06:43.731 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:06:43.732 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T09:06:43.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T09:06:43.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:06:43.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:06:43.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:06:43.751 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:06:43.750+0000 7f4c00e480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:06:43.753 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:06:43.752+0000 7f4c00e480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:06:43.755 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:06:43.754+0000 7f4c00e480c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:06:44.107 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T09:06:44.107 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:06:44.107 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:06:44.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:06:44.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:06:44.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:06:44.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:06:44.109 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:06:44.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:06:44.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:06:44.271 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:06:44.270+0000 7f4c00e480c0 -1 Falling back to public interface 2021-05-12T09:06:44.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:06:45.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:06:45.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:06:45.498 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:06:45.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:06:45.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:06:45.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:06:45.920 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:06:46.805 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:06:46.804+0000 7f4c00e480c0 -1 osd.0 32 log_to_monitors {default=true} 2021-05-12T09:06:46.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:06:46.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:06:46.922 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:06:46.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:06:46.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:06:46.924 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:06:47.350 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up out weight 0 up_from 39 up_thru 0 down_at 33 last_clean_interval [5,32) [v2:127.0.0.1:6802/49441,v1:127.0.0.1:6803/49441] [v2:127.0.0.1:6804/49441,v1:127.0.0.1:6805/49441] exists,up c5a51d7a-897f-41a2-97c3-e9ef91d860e1 2021-05-12T09:06:47.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:06:47.355 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:06:47.355 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:06:47.355 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2021-05-12T09:06:47.355 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 1 2021-05-12T09:06:47.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:06:47.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:06:47.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T09:06:47.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:06:47.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T09:06:47.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:06:47.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:06:47.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:06:47.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:06:47.358 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T09:06:47.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T09:06:47.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:06:47.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:06:47.360 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:06:47.360 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:06:47.360 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:06:47.360 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:06:47.361 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:06:47.361 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:06:47.361 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:06:47.361 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:06:47.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:06:47.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:06:47.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:06:47.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:06:47.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:06:47.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:06:47.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:06:47.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:06:47.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:06:47.364 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T09:06:47.364 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T09:06:47.364 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:06:47.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:06:47.366 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T09:06:47.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T09:06:47.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:06:47.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:06:47.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:06:47.382 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:06:47.381+0000 7f72eb6760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:06:47.383 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:06:47.383+0000 7f72eb6760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:06:47.385 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:06:47.384+0000 7f72eb6760c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:06:47.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T09:06:47.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:06:47.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:06:47.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:06:47.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:06:47.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:06:47.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:06:47.743 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:06:47.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:06:47.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:06:47.901 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:06:47.900+0000 7f72eb6760c0 -1 Falling back to public interface 2021-05-12T09:06:48.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:06:49.138 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:06:49.138 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:06:49.139 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:06:49.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:06:49.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:06:49.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:06:49.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:06:50.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:06:50.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:06:50.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:06:50.537 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:06:50.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:06:50.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:06:50.541 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:06:50.540+0000 7f72eb6760c0 -1 osd.1 32 log_to_monitors {default=true} 2021-05-12T09:06:50.952 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up out weight 0 up_from 41 up_thru 16 down_at 34 last_clean_interval [10,32) [v2:127.0.0.1:6810/49998,v1:127.0.0.1:6811/49998] [v2:127.0.0.1:6812/49998,v1:127.0.0.1:6813/49998] exists,up e003828b-f9e6-48de-bd5a-128e4af3e5dc 2021-05-12T09:06:50.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:06:50.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:06:50.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:06:50.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2021-05-12T09:06:50.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2021-05-12T09:06:51.498 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already in. 2021-05-12T09:06:51.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2021-05-12T09:06:51.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.1 2021-05-12T09:06:52.185 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already in. 2021-05-12T09:06:52.195 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T09:06:52.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2021-05-12T09:06:52.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:06:52.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:06:52.197 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:06:52.197 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:06:52.197 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T09:06:52.197 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:06:52.197 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:06:52.198 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:06:52.198 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:06:52.199 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:06:52.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:06:52.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:06:52.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:06:52.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:06:52.255 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:06:52.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:06:52.671 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:06:52.673 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:06:52.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:06:52.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:06:52.674 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:06:52.811 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=167503724548 2021-05-12T09:06:52.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 167503724548 2021-05-12T09:06:52.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-167503724548' 2021-05-12T09:06:52.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:06:52.812 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:06:52.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=176093659139 2021-05-12T09:06:52.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 176093659139 2021-05-12T09:06:52.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-167503724548 1-176093659139' 2021-05-12T09:06:52.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:06:52.952 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:06:53.056 INFO:tasks.workunit.client.0.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-05-12T09:06:53.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T09:06:53.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T09:06:53.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T09:06:53.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:06:53.059 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-167503724548 2021-05-12T09:06:53.059 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:06:53.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:06:53.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-167503724548 2021-05-12T09:06:53.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:06:53.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=167503724548 2021-05-12T09:06:53.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 167503724548' 2021-05-12T09:06:53.062 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 167503724548 2021-05-12T09:06:53.063 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:06:53.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 1 -lt 167503724548 2021-05-12T09:06:53.445 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:06:54.445 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:06:54.446 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:06:54.842 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 167503724548 -lt 167503724548 2021-05-12T09:06:54.842 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:06:54.843 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-176093659139 2021-05-12T09:06:54.843 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:06:54.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:06:54.844 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-176093659139 2021-05-12T09:06:54.844 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:06:54.845 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=176093659139 2021-05-12T09:06:54.845 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 176093659139' 2021-05-12T09:06:54.845 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 176093659139 2021-05-12T09:06:54.846 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:06:55.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 176093659139 -lt 176093659139 2021-05-12T09:06:55.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:06:55.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:06:55.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:06:55.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:06:55.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:06:55.752 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:06:55.752 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:06:55.752 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:06:55.753 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:06:55.753 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:06:55.753 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:06:56.134 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:06:56.135 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:06:56.135 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:06:56.135 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:06:56.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:06:56.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T09:06:56.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T09:06:56.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T09:06:56.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:06:56.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:06:56.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:06:56.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:06:56.746 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:06:56.747 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:06:56.747 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:06:56.747 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:06:56.748 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:06:56.748 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:06:57.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:06:57.118 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:06:57.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:06:57.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:06:57.628 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:06:57.629 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:06:57.629 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:06:57.629 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:06:57.629 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:06:57.630 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:06:57.630 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:06:57.630 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:06:57.630 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:06:58.138 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:06:58.139 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:06:58.139 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T09:06:58.139 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:06:58.139 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T09:06:58.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:06:58.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:06:58.342 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:06:58.342 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:06:58.342 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:06:58.342 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:06:58.343 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:06:58.343 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:06:58.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:06:58.719 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:06:58.719 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:06:58.719 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:06:59.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:06:59.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:06:59.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:06:59.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:06:59.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:06:59.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:06:59.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:06:59.226 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:06:59.226 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:06:59.734 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:06:59.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:06:59.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T09:06:59.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:06:59.736 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T09:07:00.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:07:00.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:07:00.137 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:07:00.137 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:07:00.139 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:07:00.139 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:07:00.139 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:07:00.139 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:07:00.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:07:00.516 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:07:00.516 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:00.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:01.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:07:01.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:07:01.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:07:01.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:07:01.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:07:01.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:07:01.029 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:07:01.030 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:07:01.030 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:07:01.534 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:07:01.534 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:07:01.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T09:07:01.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:07:01.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T09:07:02.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:07:02.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:07:02.336 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:07:02.336 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:07:02.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:07:02.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:07:02.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:07:02.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:07:02.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:07:02.723 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:07:02.724 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:02.724 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:03.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:07:03.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:07:03.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:07:03.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:07:03.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:07:03.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:07:03.229 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:07:03.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:07:03.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:07:03.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:07:03.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:07:03.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T09:07:03.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:07:03.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T09:07:05.331 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:07:05.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:07:05.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:07:05.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:07:05.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:07:05.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:07:05.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:07:05.335 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:07:05.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:07:05.722 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:07:05.723 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:05.723 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:06.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:07:06.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:07:06.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:07:06.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:07:06.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:07:06.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:07:06.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:07:06.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:07:06.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:07:06.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:07:06.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:07:06.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T09:07:06.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:07:06.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T09:07:09.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:07:09.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:07:09.929 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:07:09.929 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:07:09.930 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:07:09.930 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:07:09.930 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:07:09.931 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:07:10.321 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:07:10.321 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:07:10.322 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:10.322 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:10.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:07:10.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:07:10.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:07:10.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:07:10.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:07:10.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:07:10.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:07:10.828 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:07:10.829 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:07:11.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:07:11.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:07:11.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T09:07:11.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:07:11.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T09:07:17.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:07:17.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:07:17.739 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:07:17.739 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:07:17.740 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:07:17.740 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:07:17.740 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:07:17.740 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:07:18.130 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:07:18.130 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:07:18.130 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:18.131 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:18.636 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:07:18.636 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:07:18.636 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:07:18.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:07:18.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:07:18.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:07:18.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:07:18.638 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:07:18.638 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:07:19.138 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:07:19.138 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:07:19.139 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T09:07:19.139 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:07:19.139 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T09:07:26.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:07:26.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:07:26.441 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:07:26.441 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:07:26.441 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:07:26.441 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:07:26.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:07:26.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:07:26.826 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:07:26.827 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:07:26.828 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:26.829 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:27.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:07:27.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:07:27.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:07:27.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:07:27.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:07:27.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:07:27.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:07:27.331 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:07:27.331 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:07:27.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:07:27.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:07:27.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T09:07:27.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T09:07:28.331 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:07:28.331 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_fingerprint": "b157a664-9ad2-403b-83e9-0841dc9dbc16", 2021-05-12T09:07:28.332 INFO:tasks.workunit.client.0.smithi028.stdout: "version": "17.0.0-4095-ga8c673d5", 2021-05-12T09:07:28.332 INFO:tasks.workunit.client.0.smithi028.stdout: "commit": "a8c673d555488beec9ccda5812ba822e69c12ea9", 2021-05-12T09:07:28.332 INFO:tasks.workunit.client.0.smithi028.stdout: "timestamp": "2021-05-12T09:07:28.330341+0000", 2021-05-12T09:07:28.332 INFO:tasks.workunit.client.0.smithi028.stdout: "tag": "", 2021-05-12T09:07:28.333 INFO:tasks.workunit.client.0.smithi028.stdout: "health": { 2021-05-12T09:07:28.333 INFO:tasks.workunit.client.0.smithi028.stdout: "status": "HEALTH_WARN", 2021-05-12T09:07:28.333 INFO:tasks.workunit.client.0.smithi028.stdout: "checks": { 2021-05-12T09:07:28.334 INFO:tasks.workunit.client.0.smithi028.stdout: "OSDMAP_FLAGS": { 2021-05-12T09:07:28.334 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:07:28.334 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:07:28.334 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T09:07:28.335 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 17 2021-05-12T09:07:28.335 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.335 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [], 2021-05-12T09:07:28.335 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:07:28.336 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.336 INFO:tasks.workunit.client.0.smithi028.stdout: "OSD_DOWN": { 2021-05-12T09:07:28.336 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:07:28.336 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:07:28.336 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "1 osds down", 2021-05-12T09:07:28.337 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 1 2021-05-12T09:07:28.337 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.338 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:07:28.338 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.338 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "osd.2 (root=default,host=smithi028) is down" 2021-05-12T09:07:28.338 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.338 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.339 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:07:28.339 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.339 INFO:tasks.workunit.client.0.smithi028.stdout: "PG_DEGRADED": { 2021-05-12T09:07:28.339 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:07:28.339 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:07:28.340 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "Degraded data redundancy: 102/306 objects degraded (33.333%), 2 pgs degraded", 2021-05-12T09:07:28.340 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 2 2021-05-12T09:07:28.340 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.341 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:07:28.341 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.341 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:07:28.341 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.341 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.342 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.0 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:07:28.342 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.342 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.342 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:07:28.342 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.343 INFO:tasks.workunit.client.0.smithi028.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T09:07:28.343 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:07:28.343 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:07:28.343 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T09:07:28.344 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 1 2021-05-12T09:07:28.344 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.345 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:07:28.345 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.345 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T09:07:28.345 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.346 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.346 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T09:07:28.346 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.347 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.347 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:07:28.347 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.347 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.348 INFO:tasks.workunit.client.0.smithi028.stdout: "mutes": [] 2021-05-12T09:07:28.348 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.348 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap_first_committed": 1, 2021-05-12T09:07:28.348 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap_last_committed": 1, 2021-05-12T09:07:28.349 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap": { 2021-05-12T09:07:28.349 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 1, 2021-05-12T09:07:28.349 INFO:tasks.workunit.client.0.smithi028.stdout: "fsid": "8342ec9e-531f-44b4-8c8c-079d667fe908", 2021-05-12T09:07:28.349 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:05:27.779860Z", 2021-05-12T09:07:28.350 INFO:tasks.workunit.client.0.smithi028.stdout: "created": "2021-05-12T09:05:27.779860Z", 2021-05-12T09:07:28.350 INFO:tasks.workunit.client.0.smithi028.stdout: "min_mon_release": 17, 2021-05-12T09:07:28.350 INFO:tasks.workunit.client.0.smithi028.stdout: "min_mon_release_name": "quincy", 2021-05-12T09:07:28.350 INFO:tasks.workunit.client.0.smithi028.stdout: "election_strategy": 1, 2021-05-12T09:07:28.351 INFO:tasks.workunit.client.0.smithi028.stdout: "disallowed_leaders: ": "", 2021-05-12T09:07:28.351 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode": false, 2021-05-12T09:07:28.351 INFO:tasks.workunit.client.0.smithi028.stdout: "features": { 2021-05-12T09:07:28.351 INFO:tasks.workunit.client.0.smithi028.stdout: "persistent": [ 2021-05-12T09:07:28.352 INFO:tasks.workunit.client.0.smithi028.stdout: "kraken", 2021-05-12T09:07:28.352 INFO:tasks.workunit.client.0.smithi028.stdout: "luminous", 2021-05-12T09:07:28.352 INFO:tasks.workunit.client.0.smithi028.stdout: "mimic", 2021-05-12T09:07:28.352 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap-prune", 2021-05-12T09:07:28.352 INFO:tasks.workunit.client.0.smithi028.stdout: "nautilus", 2021-05-12T09:07:28.353 INFO:tasks.workunit.client.0.smithi028.stdout: "octopus", 2021-05-12T09:07:28.353 INFO:tasks.workunit.client.0.smithi028.stdout: "pacific", 2021-05-12T09:07:28.353 INFO:tasks.workunit.client.0.smithi028.stdout: "elector-pinging", 2021-05-12T09:07:28.353 INFO:tasks.workunit.client.0.smithi028.stdout: "quincy" 2021-05-12T09:07:28.354 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.354 INFO:tasks.workunit.client.0.smithi028.stdout: "optional": [] 2021-05-12T09:07:28.354 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.355 INFO:tasks.workunit.client.0.smithi028.stdout: "mons": [ 2021-05-12T09:07:28.355 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.355 INFO:tasks.workunit.client.0.smithi028.stdout: "rank": 0, 2021-05-12T09:07:28.356 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "a", 2021-05-12T09:07:28.356 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:07:28.356 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:07:28.357 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.357 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:07:28.358 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:7115", 2021-05-12T09:07:28.358 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 0 2021-05-12T09:07:28.358 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.359 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.359 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.359 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T09:07:28.359 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T09:07:28.360 INFO:tasks.workunit.client.0.smithi028.stdout: "priority": 0, 2021-05-12T09:07:28.360 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 0, 2021-05-12T09:07:28.360 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_location": "{}" 2021-05-12T09:07:28.360 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.361 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.361 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.361 INFO:tasks.workunit.client.0.smithi028.stdout: "quorum": [ 2021-05-12T09:07:28.361 INFO:tasks.workunit.client.0.smithi028.stdout: 0 2021-05-12T09:07:28.361 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.362 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap": { 2021-05-12T09:07:28.362 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 47, 2021-05-12T09:07:28.362 INFO:tasks.workunit.client.0.smithi028.stdout: "fsid": "8342ec9e-531f-44b4-8c8c-079d667fe908", 2021-05-12T09:07:28.363 INFO:tasks.workunit.client.0.smithi028.stdout: "created": "2021-05-12T09:05:28.578482+0000", 2021-05-12T09:07:28.363 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:06:52.602275+0000", 2021-05-12T09:07:28.363 INFO:tasks.workunit.client.0.smithi028.stdout: "last_up_change": "2021-05-12T09:06:50.813501+0000", 2021-05-12T09:07:28.363 INFO:tasks.workunit.client.0.smithi028.stdout: "last_in_change": "2021-05-12T09:06:51.889810+0000", 2021-05-12T09:07:28.364 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T09:07:28.364 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_num": 5800384, 2021-05-12T09:07:28.364 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_set": [ 2021-05-12T09:07:28.365 INFO:tasks.workunit.client.0.smithi028.stdout: "nodown", 2021-05-12T09:07:28.365 INFO:tasks.workunit.client.0.smithi028.stdout: "noin", 2021-05-12T09:07:28.365 INFO:tasks.workunit.client.0.smithi028.stdout: "noout", 2021-05-12T09:07:28.365 INFO:tasks.workunit.client.0.smithi028.stdout: "pglog_hardlimit", 2021-05-12T09:07:28.366 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snapdirs", 2021-05-12T09:07:28.366 INFO:tasks.workunit.client.0.smithi028.stdout: "recovery_deletes", 2021-05-12T09:07:28.366 INFO:tasks.workunit.client.0.smithi028.stdout: "sortbitwise" 2021-05-12T09:07:28.367 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.367 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_version": 9, 2021-05-12T09:07:28.367 INFO:tasks.workunit.client.0.smithi028.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T09:07:28.367 INFO:tasks.workunit.client.0.smithi028.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T09:07:28.368 INFO:tasks.workunit.client.0.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T09:07:28.368 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_snapshot": "", 2021-05-12T09:07:28.368 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_max": 2, 2021-05-12T09:07:28.368 INFO:tasks.workunit.client.0.smithi028.stdout: "max_osd": 3, 2021-05-12T09:07:28.368 INFO:tasks.workunit.client.0.smithi028.stdout: "require_min_compat_client": "luminous", 2021-05-12T09:07:28.369 INFO:tasks.workunit.client.0.smithi028.stdout: "min_compat_client": "jewel", 2021-05-12T09:07:28.369 INFO:tasks.workunit.client.0.smithi028.stdout: "require_osd_release": "quincy", 2021-05-12T09:07:28.369 INFO:tasks.workunit.client.0.smithi028.stdout: "pools": [ 2021-05-12T09:07:28.369 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.369 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 1, 2021-05-12T09:07:28.370 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_name": ".mgr", 2021-05-12T09:07:28.370 INFO:tasks.workunit.client.0.smithi028.stdout: "create_time": "2021-05-12T09:05:55.843224+0000", 2021-05-12T09:07:28.370 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": 1, 2021-05-12T09:07:28.370 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_names": "hashpspool", 2021-05-12T09:07:28.370 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:07:28.371 INFO:tasks.workunit.client.0.smithi028.stdout: "size": 3, 2021-05-12T09:07:28.371 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 2, 2021-05-12T09:07:28.371 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_rule": 0, 2021-05-12T09:07:28.371 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_count": 0, 2021-05-12T09:07:28.372 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_target": 0, 2021-05-12T09:07:28.372 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T09:07:28.372 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T09:07:28.372 INFO:tasks.workunit.client.0.smithi028.stdout: "object_hash": 2, 2021-05-12T09:07:28.372 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_autoscale_mode": "off", 2021-05-12T09:07:28.373 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num": 1, 2021-05-12T09:07:28.373 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num": 1, 2021-05-12T09:07:28.373 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num_target": 1, 2021-05-12T09:07:28.373 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_target": 1, 2021-05-12T09:07:28.373 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_pending": 1, 2021-05-12T09:07:28.374 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pg_merge_meta": { 2021-05-12T09:07:28.374 INFO:tasks.workunit.client.0.smithi028.stdout: "source_pgid": "0.0", 2021-05-12T09:07:28.374 INFO:tasks.workunit.client.0.smithi028.stdout: "ready_epoch": 0, 2021-05-12T09:07:28.374 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_started": 0, 2021-05-12T09:07:28.375 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": 0, 2021-05-12T09:07:28.375 INFO:tasks.workunit.client.0.smithi028.stdout: "source_version": "0'0", 2021-05-12T09:07:28.375 INFO:tasks.workunit.client.0.smithi028.stdout: "target_version": "0'0" 2021-05-12T09:07:28.375 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.375 INFO:tasks.workunit.client.0.smithi028.stdout: "last_change": "21", 2021-05-12T09:07:28.376 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend": "0", 2021-05-12T09:07:28.376 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T09:07:28.376 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T09:07:28.376 INFO:tasks.workunit.client.0.smithi028.stdout: "auid": 0, 2021-05-12T09:07:28.376 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_mode": "selfmanaged", 2021-05-12T09:07:28.377 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_seq": 0, 2021-05-12T09:07:28.377 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_epoch": 0, 2021-05-12T09:07:28.377 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_snaps": [], 2021-05-12T09:07:28.377 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps": "[]", 2021-05-12T09:07:28.378 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_bytes": 0, 2021-05-12T09:07:28.378 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_objects": 0, 2021-05-12T09:07:28.378 INFO:tasks.workunit.client.0.smithi028.stdout: "tiers": [], 2021-05-12T09:07:28.378 INFO:tasks.workunit.client.0.smithi028.stdout: "tier_of": -1, 2021-05-12T09:07:28.378 INFO:tasks.workunit.client.0.smithi028.stdout: "read_tier": -1, 2021-05-12T09:07:28.379 INFO:tasks.workunit.client.0.smithi028.stdout: "write_tier": -1, 2021-05-12T09:07:28.379 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_mode": "none", 2021-05-12T09:07:28.379 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_bytes": 0, 2021-05-12T09:07:28.379 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_objects": 0, 2021-05-12T09:07:28.380 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T09:07:28.380 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T09:07:28.380 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T09:07:28.380 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_flush_age": 0, 2021-05-12T09:07:28.380 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_evict_age": 0, 2021-05-12T09:07:28.381 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profile": "", 2021-05-12T09:07:28.381 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_params": { 2021-05-12T09:07:28.381 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "none" 2021-05-12T09:07:28.381 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.381 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_period": 0, 2021-05-12T09:07:28.382 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_count": 0, 2021-05-12T09:07:28.382 INFO:tasks.workunit.client.0.smithi028.stdout: "use_gmt_hitset": true, 2021-05-12T09:07:28.382 INFO:tasks.workunit.client.0.smithi028.stdout: "min_read_recency_for_promote": 0, 2021-05-12T09:07:28.382 INFO:tasks.workunit.client.0.smithi028.stdout: "min_write_recency_for_promote": 0, 2021-05-12T09:07:28.382 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T09:07:28.383 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_search_last_n": 0, 2021-05-12T09:07:28.383 INFO:tasks.workunit.client.0.smithi028.stdout: "grade_table": [], 2021-05-12T09:07:28.383 INFO:tasks.workunit.client.0.smithi028.stdout: "stripe_width": 0, 2021-05-12T09:07:28.383 INFO:tasks.workunit.client.0.smithi028.stdout: "expected_num_objects": 0, 2021-05-12T09:07:28.384 INFO:tasks.workunit.client.0.smithi028.stdout: "fast_read": false, 2021-05-12T09:07:28.384 INFO:tasks.workunit.client.0.smithi028.stdout: "options": { 2021-05-12T09:07:28.384 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_min": 1 2021-05-12T09:07:28.384 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.384 INFO:tasks.workunit.client.0.smithi028.stdout: "application_metadata": { 2021-05-12T09:07:28.385 INFO:tasks.workunit.client.0.smithi028.stdout: "mgr": {} 2021-05-12T09:07:28.385 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.385 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.385 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.385 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 2, 2021-05-12T09:07:28.386 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_name": "test", 2021-05-12T09:07:28.386 INFO:tasks.workunit.client.0.smithi028.stdout: "create_time": "2021-05-12T09:05:59.272553+0000", 2021-05-12T09:07:28.386 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": 1, 2021-05-12T09:07:28.386 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_names": "hashpspool", 2021-05-12T09:07:28.386 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:07:28.387 INFO:tasks.workunit.client.0.smithi028.stdout: "size": 3, 2021-05-12T09:07:28.387 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 2, 2021-05-12T09:07:28.387 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_rule": 0, 2021-05-12T09:07:28.387 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_count": 0, 2021-05-12T09:07:28.387 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_target": 0, 2021-05-12T09:07:28.388 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T09:07:28.388 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T09:07:28.388 INFO:tasks.workunit.client.0.smithi028.stdout: "object_hash": 2, 2021-05-12T09:07:28.388 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_autoscale_mode": "off", 2021-05-12T09:07:28.389 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num": 1, 2021-05-12T09:07:28.389 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num": 1, 2021-05-12T09:07:28.389 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num_target": 1, 2021-05-12T09:07:28.389 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_target": 1, 2021-05-12T09:07:28.389 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_pending": 1, 2021-05-12T09:07:28.390 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pg_merge_meta": { 2021-05-12T09:07:28.390 INFO:tasks.workunit.client.0.smithi028.stdout: "source_pgid": "0.0", 2021-05-12T09:07:28.390 INFO:tasks.workunit.client.0.smithi028.stdout: "ready_epoch": 0, 2021-05-12T09:07:28.390 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_started": 0, 2021-05-12T09:07:28.390 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": 0, 2021-05-12T09:07:28.391 INFO:tasks.workunit.client.0.smithi028.stdout: "source_version": "0'0", 2021-05-12T09:07:28.391 INFO:tasks.workunit.client.0.smithi028.stdout: "target_version": "0'0" 2021-05-12T09:07:28.391 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.391 INFO:tasks.workunit.client.0.smithi028.stdout: "last_change": "32", 2021-05-12T09:07:28.392 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend": "0", 2021-05-12T09:07:28.392 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T09:07:28.392 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T09:07:28.392 INFO:tasks.workunit.client.0.smithi028.stdout: "auid": 0, 2021-05-12T09:07:28.392 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_mode": "selfmanaged", 2021-05-12T09:07:28.393 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_seq": 0, 2021-05-12T09:07:28.393 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_epoch": 0, 2021-05-12T09:07:28.393 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_snaps": [], 2021-05-12T09:07:28.393 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps": "[]", 2021-05-12T09:07:28.393 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_bytes": 0, 2021-05-12T09:07:28.394 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_objects": 0, 2021-05-12T09:07:28.394 INFO:tasks.workunit.client.0.smithi028.stdout: "tiers": [], 2021-05-12T09:07:28.394 INFO:tasks.workunit.client.0.smithi028.stdout: "tier_of": -1, 2021-05-12T09:07:28.394 INFO:tasks.workunit.client.0.smithi028.stdout: "read_tier": -1, 2021-05-12T09:07:28.395 INFO:tasks.workunit.client.0.smithi028.stdout: "write_tier": -1, 2021-05-12T09:07:28.395 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_mode": "none", 2021-05-12T09:07:28.395 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_bytes": 0, 2021-05-12T09:07:28.395 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_objects": 0, 2021-05-12T09:07:28.395 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T09:07:28.396 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T09:07:28.396 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T09:07:28.396 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_flush_age": 0, 2021-05-12T09:07:28.396 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_evict_age": 0, 2021-05-12T09:07:28.396 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profile": "", 2021-05-12T09:07:28.397 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_params": { 2021-05-12T09:07:28.397 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "none" 2021-05-12T09:07:28.397 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.397 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_period": 0, 2021-05-12T09:07:28.397 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_count": 0, 2021-05-12T09:07:28.398 INFO:tasks.workunit.client.0.smithi028.stdout: "use_gmt_hitset": true, 2021-05-12T09:07:28.398 INFO:tasks.workunit.client.0.smithi028.stdout: "min_read_recency_for_promote": 0, 2021-05-12T09:07:28.398 INFO:tasks.workunit.client.0.smithi028.stdout: "min_write_recency_for_promote": 0, 2021-05-12T09:07:28.398 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T09:07:28.398 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_search_last_n": 0, 2021-05-12T09:07:28.399 INFO:tasks.workunit.client.0.smithi028.stdout: "grade_table": [], 2021-05-12T09:07:28.399 INFO:tasks.workunit.client.0.smithi028.stdout: "stripe_width": 0, 2021-05-12T09:07:28.399 INFO:tasks.workunit.client.0.smithi028.stdout: "expected_num_objects": 0, 2021-05-12T09:07:28.399 INFO:tasks.workunit.client.0.smithi028.stdout: "fast_read": false, 2021-05-12T09:07:28.400 INFO:tasks.workunit.client.0.smithi028.stdout: "options": {}, 2021-05-12T09:07:28.400 INFO:tasks.workunit.client.0.smithi028.stdout: "application_metadata": {} 2021-05-12T09:07:28.400 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.400 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.400 INFO:tasks.workunit.client.0.smithi028.stdout: "osds": [ 2021-05-12T09:07:28.401 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.401 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:07:28.401 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "c5a51d7a-897f-41a2-97c3-e9ef91d860e1", 2021-05-12T09:07:28.401 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 1, 2021-05-12T09:07:28.401 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:07:28.402 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:07:28.402 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:07:28.402 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 5, 2021-05-12T09:07:28.402 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 32, 2021-05-12T09:07:28.403 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 39, 2021-05-12T09:07:28.403 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 45, 2021-05-12T09:07:28.403 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 33, 2021-05-12T09:07:28.403 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:07:28.403 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:07:28.404 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:07:28.404 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.404 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:07:28.404 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6802", 2021-05-12T09:07:28.404 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:07:28.405 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.405 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.405 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:07:28.405 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6803", 2021-05-12T09:07:28.406 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:07:28.406 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.406 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.406 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.406 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:07:28.407 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:07:28.407 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.407 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:07:28.407 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6804", 2021-05-12T09:07:28.407 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:07:28.408 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.408 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.408 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:07:28.408 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6805", 2021-05-12T09:07:28.408 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:07:28.409 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.409 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.409 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.409 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:07:28.409 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:07:28.410 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.410 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:07:28.410 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6808", 2021-05-12T09:07:28.410 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:07:28.411 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.411 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.411 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:07:28.411 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6809", 2021-05-12T09:07:28.411 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:07:28.412 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.412 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.412 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.412 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:07:28.412 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:07:28.413 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.413 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:07:28.413 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6806", 2021-05-12T09:07:28.413 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:07:28.413 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.414 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.414 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:07:28.414 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6807", 2021-05-12T09:07:28.414 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:07:28.415 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.415 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.415 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.415 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6803/49441", 2021-05-12T09:07:28.415 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6805/49441", 2021-05-12T09:07:28.416 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6809/49441", 2021-05-12T09:07:28.416 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6807/49441", 2021-05-12T09:07:28.416 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:07:28.416 INFO:tasks.workunit.client.0.smithi028.stdout: "exists", 2021-05-12T09:07:28.416 INFO:tasks.workunit.client.0.smithi028.stdout: "up" 2021-05-12T09:07:28.417 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.417 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.417 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.417 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:07:28.417 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "e003828b-f9e6-48de-bd5a-128e4af3e5dc", 2021-05-12T09:07:28.418 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 1, 2021-05-12T09:07:28.418 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:07:28.418 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:07:28.418 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:07:28.419 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 10, 2021-05-12T09:07:28.419 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 32, 2021-05-12T09:07:28.419 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 41, 2021-05-12T09:07:28.419 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 46, 2021-05-12T09:07:28.420 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 34, 2021-05-12T09:07:28.420 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:07:28.420 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:07:28.420 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:07:28.420 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.421 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:07:28.421 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6810", 2021-05-12T09:07:28.421 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:07:28.421 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.421 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.422 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:07:28.422 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6811", 2021-05-12T09:07:28.422 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:07:28.422 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.422 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.423 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.423 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:07:28.423 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:07:28.423 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.423 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:07:28.424 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6812", 2021-05-12T09:07:28.424 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:07:28.424 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.424 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.425 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:07:28.425 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6813", 2021-05-12T09:07:28.425 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:07:28.425 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.425 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.426 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.426 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:07:28.426 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:07:28.426 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.426 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:07:28.427 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6816", 2021-05-12T09:07:28.427 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:07:28.427 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.427 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.427 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:07:28.428 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6817", 2021-05-12T09:07:28.428 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:07:28.428 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.428 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.428 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.429 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:07:28.429 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:07:28.429 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.429 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:07:28.430 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6814", 2021-05-12T09:07:28.430 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:07:28.430 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.430 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.430 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:07:28.431 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6815", 2021-05-12T09:07:28.431 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:07:28.431 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.431 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.431 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.432 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6811/49998", 2021-05-12T09:07:28.432 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6813/49998", 2021-05-12T09:07:28.432 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6817/49998", 2021-05-12T09:07:28.432 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6815/49998", 2021-05-12T09:07:28.432 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:07:28.433 INFO:tasks.workunit.client.0.smithi028.stdout: "exists", 2021-05-12T09:07:28.433 INFO:tasks.workunit.client.0.smithi028.stdout: "up" 2021-05-12T09:07:28.433 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.433 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.433 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.434 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:07:28.434 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "10e9d857-0855-44ba-8b8e-8d823ab9204a", 2021-05-12T09:07:28.434 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 0, 2021-05-12T09:07:28.434 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:07:28.435 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:07:28.435 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:07:28.435 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 0, 2021-05-12T09:07:28.435 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 0, 2021-05-12T09:07:28.435 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 15, 2021-05-12T09:07:28.436 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 26, 2021-05-12T09:07:28.436 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 35, 2021-05-12T09:07:28.436 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:07:28.436 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:07:28.436 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:07:28.437 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.437 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:07:28.437 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6818", 2021-05-12T09:07:28.437 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:07:28.437 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.438 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.438 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:07:28.438 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6819", 2021-05-12T09:07:28.438 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:07:28.438 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.439 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.439 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.439 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:07:28.439 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:07:28.440 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.440 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:07:28.440 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6820", 2021-05-12T09:07:28.440 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:07:28.440 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.441 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.441 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:07:28.441 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6821", 2021-05-12T09:07:28.441 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:07:28.441 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.442 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.442 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.442 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:07:28.442 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:07:28.442 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.443 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:07:28.443 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6824", 2021-05-12T09:07:28.443 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:07:28.443 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.443 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.444 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:07:28.444 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6825", 2021-05-12T09:07:28.444 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:07:28.444 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.445 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.445 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.445 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:07:28.445 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:07:28.445 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.446 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:07:28.446 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6822", 2021-05-12T09:07:28.446 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:07:28.446 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.446 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.447 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:07:28.447 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6823", 2021-05-12T09:07:28.447 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:07:28.447 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.447 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.448 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.448 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6819/45341", 2021-05-12T09:07:28.448 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6821/45341", 2021-05-12T09:07:28.448 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6825/45341", 2021-05-12T09:07:28.449 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6823/45341", 2021-05-12T09:07:28.449 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:07:28.449 INFO:tasks.workunit.client.0.smithi028.stdout: "exists" 2021-05-12T09:07:28.449 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.449 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.450 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.450 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_xinfo": [ 2021-05-12T09:07:28.450 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.450 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:07:28.450 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:06:40.977681+0000", 2021-05-12T09:07:28.451 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:07:28.451 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:07:28.451 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:07:28.451 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:07:28.451 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:05:37.831220+0000", 2021-05-12T09:07:28.452 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:07:28.452 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.452 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.452 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:07:28.453 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:06:41.506912+0000", 2021-05-12T09:07:28.453 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:07:28.453 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:07:28.453 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:07:28.453 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:07:28.454 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:05:46.839891+0000", 2021-05-12T09:07:28.454 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:07:28.454 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.454 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.454 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:07:28.455 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:06:42.039554+0000", 2021-05-12T09:07:28.455 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:07:28.455 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:07:28.455 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:07:28.455 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:07:28.456 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:05:56.142117+0000", 2021-05-12T09:07:28.456 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:07:28.456 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.456 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.456 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_upmap": [], 2021-05-12T09:07:28.457 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_upmap_items": [], 2021-05-12T09:07:28.457 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_temp": [], 2021-05-12T09:07:28.457 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_temp": [], 2021-05-12T09:07:28.457 INFO:tasks.workunit.client.0.smithi028.stdout: "blocklist": {}, 2021-05-12T09:07:28.458 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profiles": { 2021-05-12T09:07:28.458 INFO:tasks.workunit.client.0.smithi028.stdout: "default": { 2021-05-12T09:07:28.458 INFO:tasks.workunit.client.0.smithi028.stdout: "k": "2", 2021-05-12T09:07:28.458 INFO:tasks.workunit.client.0.smithi028.stdout: "m": "2", 2021-05-12T09:07:28.458 INFO:tasks.workunit.client.0.smithi028.stdout: "plugin": "jerasure", 2021-05-12T09:07:28.459 INFO:tasks.workunit.client.0.smithi028.stdout: "technique": "reed_sol_van" 2021-05-12T09:07:28.459 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.459 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.459 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps_queue": [], 2021-05-12T09:07:28.459 INFO:tasks.workunit.client.0.smithi028.stdout: "new_removed_snaps": [], 2021-05-12T09:07:28.460 INFO:tasks.workunit.client.0.smithi028.stdout: "new_purged_snaps": [], 2021-05-12T09:07:28.460 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_node_flags": {}, 2021-05-12T09:07:28.460 INFO:tasks.workunit.client.0.smithi028.stdout: "device_class_flags": {}, 2021-05-12T09:07:28.460 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode": { 2021-05-12T09:07:28.460 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode_enabled": false, 2021-05-12T09:07:28.461 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_bucket_count": 0, 2021-05-12T09:07:28.461 INFO:tasks.workunit.client.0.smithi028.stdout: "degraded_stretch_mode": 0, 2021-05-12T09:07:28.461 INFO:tasks.workunit.client.0.smithi028.stdout: "recovering_stretch_mode": 0, 2021-05-12T09:07:28.461 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode_bucket": 0 2021-05-12T09:07:28.461 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.462 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.462 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_metadata": [ 2021-05-12T09:07:28.462 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.462 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:07:28.463 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:07:28.463 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6804/49441,v1:127.0.0.1:6805/49441]", 2021-05-12T09:07:28.463 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:07:28.463 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:07:28.463 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:07:28.464 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:07:28.464 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:07:28.464 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:07:28.464 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:07:28.464 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:07:28.465 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:07:28.465 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T09:07:28.465 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:07:28.465 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:07:28.465 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:07:28.466 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:07:28.466 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:07:28.466 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:07:28.466 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:07:28.466 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:07:28.467 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:07:28.467 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:07:28.467 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:07:28.467 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:07:28.467 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:07:28.468 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:07:28.468 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:07:28.468 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6802/49441,v1:127.0.0.1:6803/49441]", 2021-05-12T09:07:28.468 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:07:28.469 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/49441,v1:127.0.0.1:6809/49441]", 2021-05-12T09:07:28.469 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/49441,v1:127.0.0.1:6807/49441]", 2021-05-12T09:07:28.469 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:07:28.469 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:07:28.469 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:07:28.470 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:07:28.470 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:07:28.470 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:07:28.470 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:07:28.470 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:07:28.471 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:07:28.471 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T09:07:28.471 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:07:28.471 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:07:28.471 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:07:28.472 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.472 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.472 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:07:28.472 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:07:28.472 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6812/49998,v1:127.0.0.1:6813/49998]", 2021-05-12T09:07:28.473 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:07:28.473 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:07:28.473 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:07:28.473 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:07:28.473 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:07:28.474 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:07:28.474 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:07:28.474 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:07:28.474 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:07:28.475 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T09:07:28.475 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:07:28.475 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:07:28.475 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:07:28.475 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:07:28.476 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:07:28.476 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:07:28.476 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:07:28.476 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:07:28.476 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:07:28.477 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:07:28.477 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:07:28.477 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:07:28.477 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:07:28.477 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:07:28.478 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:07:28.478 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6810/49998,v1:127.0.0.1:6811/49998]", 2021-05-12T09:07:28.478 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:07:28.478 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/49998,v1:127.0.0.1:6817/49998]", 2021-05-12T09:07:28.478 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/49998,v1:127.0.0.1:6815/49998]", 2021-05-12T09:07:28.479 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:07:28.479 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:07:28.479 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:07:28.479 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:07:28.480 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:07:28.480 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:07:28.480 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:07:28.480 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:07:28.480 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:07:28.481 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T09:07:28.481 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:07:28.481 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:07:28.481 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:07:28.481 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.482 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.482 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:07:28.482 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:07:28.482 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6820/45341,v1:127.0.0.1:6821/45341]", 2021-05-12T09:07:28.482 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:07:28.483 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:07:28.483 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:07:28.483 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:07:28.483 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:07:28.483 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:07:28.484 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:07:28.484 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:07:28.484 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:07:28.484 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T09:07:28.485 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:07:28.485 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:07:28.485 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:07:28.485 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:07:28.485 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:07:28.486 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:07:28.486 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:07:28.486 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:07:28.486 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:07:28.486 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:07:28.487 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:07:28.487 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:07:28.487 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:07:28.487 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:07:28.487 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:07:28.488 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6818/45341,v1:127.0.0.1:6819/45341]", 2021-05-12T09:07:28.488 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:07:28.488 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/45341,v1:127.0.0.1:6825/45341]", 2021-05-12T09:07:28.488 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/45341,v1:127.0.0.1:6823/45341]", 2021-05-12T09:07:28.488 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:07:28.489 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:07:28.489 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:07:28.489 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:07:28.489 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:07:28.490 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:07:28.490 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:07:28.490 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:07:28.490 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:07:28.490 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T09:07:28.491 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:07:28.491 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:07:28.491 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:07:28.491 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.491 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.492 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_clean_epochs": { 2021-05-12T09:07:28.492 INFO:tasks.workunit.client.0.smithi028.stdout: "min_last_epoch_clean": 0, 2021-05-12T09:07:28.492 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": { 2021-05-12T09:07:28.492 INFO:tasks.workunit.client.0.smithi028.stdout: "per_pool": [] 2021-05-12T09:07:28.492 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.493 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_epochs": [ 2021-05-12T09:07:28.493 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.493 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:07:28.493 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 39 2021-05-12T09:07:28.493 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.494 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.494 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:07:28.494 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 42 2021-05-12T09:07:28.494 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.495 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.495 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:07:28.495 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 15 2021-05-12T09:07:28.495 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.495 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.496 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.496 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_first_committed": 1, 2021-05-12T09:07:28.496 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_last_committed": 47, 2021-05-12T09:07:28.496 INFO:tasks.workunit.client.0.smithi028.stdout: "crushmap": { 2021-05-12T09:07:28.496 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": [ 2021-05-12T09:07:28.497 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.497 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:07:28.497 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.0", 2021-05-12T09:07:28.497 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:07:28.497 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.498 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.498 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:07:28.498 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.1", 2021-05-12T09:07:28.498 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:07:28.498 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.499 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.499 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:07:28.499 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.2", 2021-05-12T09:07:28.499 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:07:28.500 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.500 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.500 INFO:tasks.workunit.client.0.smithi028.stdout: "types": [ 2021-05-12T09:07:28.500 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.500 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 0, 2021-05-12T09:07:28.501 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd" 2021-05-12T09:07:28.501 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.501 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.501 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:07:28.501 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "host" 2021-05-12T09:07:28.502 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.502 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.502 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 2, 2021-05-12T09:07:28.502 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "chassis" 2021-05-12T09:07:28.502 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.503 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.503 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 3, 2021-05-12T09:07:28.503 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "rack" 2021-05-12T09:07:28.503 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.503 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.504 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 4, 2021-05-12T09:07:28.504 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "row" 2021-05-12T09:07:28.504 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.504 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.504 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 5, 2021-05-12T09:07:28.505 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "pdu" 2021-05-12T09:07:28.505 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.505 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.505 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 6, 2021-05-12T09:07:28.505 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "pod" 2021-05-12T09:07:28.506 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.506 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.506 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 7, 2021-05-12T09:07:28.506 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "room" 2021-05-12T09:07:28.507 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.507 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.507 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 8, 2021-05-12T09:07:28.507 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "datacenter" 2021-05-12T09:07:28.507 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.508 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.508 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 9, 2021-05-12T09:07:28.508 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "zone" 2021-05-12T09:07:28.508 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.508 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.509 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 10, 2021-05-12T09:07:28.509 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "region" 2021-05-12T09:07:28.509 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.509 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.509 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:07:28.510 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "root" 2021-05-12T09:07:28.510 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.510 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.510 INFO:tasks.workunit.client.0.smithi028.stdout: "buckets": [ 2021-05-12T09:07:28.510 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.511 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -1, 2021-05-12T09:07:28.511 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "default", 2021-05-12T09:07:28.511 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:07:28.511 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "root", 2021-05-12T09:07:28.512 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:07:28.512 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:07:28.512 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:07:28.512 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:07:28.512 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.513 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -3, 2021-05-12T09:07:28.513 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:07:28.513 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:07:28.513 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.513 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.514 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.514 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.514 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -2, 2021-05-12T09:07:28.514 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "default~hdd", 2021-05-12T09:07:28.515 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:07:28.515 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "root", 2021-05-12T09:07:28.515 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:07:28.515 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:07:28.515 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:07:28.516 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:07:28.516 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.516 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -4, 2021-05-12T09:07:28.516 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:07:28.516 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:07:28.517 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.517 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.517 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.517 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.517 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -3, 2021-05-12T09:07:28.518 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "smithi028", 2021-05-12T09:07:28.518 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:07:28.518 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "host", 2021-05-12T09:07:28.518 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:07:28.519 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:07:28.519 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:07:28.519 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:07:28.519 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.520 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:07:28.520 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:07:28.520 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:07:28.520 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.521 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.521 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:07:28.521 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:07:28.521 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 1 2021-05-12T09:07:28.521 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.522 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.522 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:07:28.522 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:07:28.522 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 2 2021-05-12T09:07:28.522 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.523 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.523 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.523 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.523 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -4, 2021-05-12T09:07:28.523 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "smithi028~hdd", 2021-05-12T09:07:28.524 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:07:28.524 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "host", 2021-05-12T09:07:28.524 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:07:28.525 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:07:28.526 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:07:28.526 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:07:28.526 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.526 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:07:28.527 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:07:28.527 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:07:28.527 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.527 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.527 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:07:28.528 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:07:28.528 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 1 2021-05-12T09:07:28.528 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.528 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.528 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:07:28.529 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:07:28.529 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 2 2021-05-12T09:07:28.529 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.529 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.529 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.530 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.530 INFO:tasks.workunit.client.0.smithi028.stdout: "rules": [ 2021-05-12T09:07:28.530 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.530 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_id": 0, 2021-05-12T09:07:28.531 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_name": "replicated_rule", 2021-05-12T09:07:28.531 INFO:tasks.workunit.client.0.smithi028.stdout: "ruleset": 0, 2021-05-12T09:07:28.531 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:07:28.531 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 1, 2021-05-12T09:07:28.531 INFO:tasks.workunit.client.0.smithi028.stdout: "max_size": 10, 2021-05-12T09:07:28.532 INFO:tasks.workunit.client.0.smithi028.stdout: "steps": [ 2021-05-12T09:07:28.532 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.532 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "take", 2021-05-12T09:07:28.532 INFO:tasks.workunit.client.0.smithi028.stdout: "item": -1, 2021-05-12T09:07:28.532 INFO:tasks.workunit.client.0.smithi028.stdout: "item_name": "default" 2021-05-12T09:07:28.533 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.533 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.533 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "choose_firstn", 2021-05-12T09:07:28.533 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 0, 2021-05-12T09:07:28.533 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "osd" 2021-05-12T09:07:28.534 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.534 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.534 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "emit" 2021-05-12T09:07:28.534 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.534 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:07:28.535 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.535 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.535 INFO:tasks.workunit.client.0.smithi028.stdout: "tunables": { 2021-05-12T09:07:28.535 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_local_tries": 0, 2021-05-12T09:07:28.536 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_local_fallback_tries": 0, 2021-05-12T09:07:28.536 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_total_tries": 50, 2021-05-12T09:07:28.536 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_descend_once": 1, 2021-05-12T09:07:28.536 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_vary_r": 1, 2021-05-12T09:07:28.536 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_stable": 1, 2021-05-12T09:07:28.537 INFO:tasks.workunit.client.0.smithi028.stdout: "straw_calc_version": 1, 2021-05-12T09:07:28.537 INFO:tasks.workunit.client.0.smithi028.stdout: "allowed_bucket_algs": 54, 2021-05-12T09:07:28.537 INFO:tasks.workunit.client.0.smithi028.stdout: "profile": "jewel", 2021-05-12T09:07:28.537 INFO:tasks.workunit.client.0.smithi028.stdout: "optimal_tunables": 1, 2021-05-12T09:07:28.537 INFO:tasks.workunit.client.0.smithi028.stdout: "legacy_tunables": 0, 2021-05-12T09:07:28.538 INFO:tasks.workunit.client.0.smithi028.stdout: "minimum_required_version": "jewel", 2021-05-12T09:07:28.538 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables": 1, 2021-05-12T09:07:28.538 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables2": 1, 2021-05-12T09:07:28.538 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v2_rules": 0, 2021-05-12T09:07:28.538 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables3": 1, 2021-05-12T09:07:28.539 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v3_rules": 0, 2021-05-12T09:07:28.539 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v4_buckets": 1, 2021-05-12T09:07:28.539 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables5": 1, 2021-05-12T09:07:28.539 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v5_rules": 0 2021-05-12T09:07:28.539 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.540 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_args": {} 2021-05-12T09:07:28.540 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.540 INFO:tasks.workunit.client.0.smithi028.stdout: "fsmap": { 2021-05-12T09:07:28.540 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 1, 2021-05-12T09:07:28.541 INFO:tasks.workunit.client.0.smithi028.stdout: "default_fscid": -1, 2021-05-12T09:07:28.541 INFO:tasks.workunit.client.0.smithi028.stdout: "compat": { 2021-05-12T09:07:28.541 INFO:tasks.workunit.client.0.smithi028.stdout: "compat": {}, 2021-05-12T09:07:28.541 INFO:tasks.workunit.client.0.smithi028.stdout: "ro_compat": {}, 2021-05-12T09:07:28.541 INFO:tasks.workunit.client.0.smithi028.stdout: "incompat": { 2021-05-12T09:07:28.542 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_1": "base v0.20", 2021-05-12T09:07:28.542 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_2": "client writeable ranges", 2021-05-12T09:07:28.542 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T09:07:28.542 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_4": "dir inode in separate object", 2021-05-12T09:07:28.542 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T09:07:28.543 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T09:07:28.543 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_8": "no anchor table", 2021-05-12T09:07:28.543 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_9": "file layout v2", 2021-05-12T09:07:28.543 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_10": "snaprealm v2" 2021-05-12T09:07:28.543 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.544 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.544 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_flags": { 2021-05-12T09:07:28.544 INFO:tasks.workunit.client.0.smithi028.stdout: "enable_multiple": true, 2021-05-12T09:07:28.544 INFO:tasks.workunit.client.0.smithi028.stdout: "ever_enabled_multiple": true 2021-05-12T09:07:28.544 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.545 INFO:tasks.workunit.client.0.smithi028.stdout: "standbys": [], 2021-05-12T09:07:28.545 INFO:tasks.workunit.client.0.smithi028.stdout: "filesystems": [] 2021-05-12T09:07:28.545 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.545 INFO:tasks.workunit.client.0.smithi028.stdout: "mdsmap_first_committed": 1, 2021-05-12T09:07:28.545 INFO:tasks.workunit.client.0.smithi028.stdout: "mdsmap_last_committed": 1, 2021-05-12T09:07:28.546 INFO:tasks.workunit.client.0.smithi028.stdout: "auth": { 2021-05-12T09:07:28.546 INFO:tasks.workunit.client.0.smithi028.stdout: "first_committed": 1, 2021-05-12T09:07:28.546 INFO:tasks.workunit.client.0.smithi028.stdout: "last_committed": 5, 2021-05-12T09:07:28.546 INFO:tasks.workunit.client.0.smithi028.stdout: "num_secrets": 3 2021-05-12T09:07:28.547 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.547 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 2, 2021-05-12T09:07:28.547 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_active": 2, 2021-05-12T09:07:28.547 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_unknown": 0, 2021-05-12T09:07:28.547 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osd": 3, 2021-05-12T09:07:28.548 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_sum": { 2021-05-12T09:07:28.548 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:07:28.548 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 507580, 2021-05-12T09:07:28.548 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 102, 2021-05-12T09:07:28.548 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:07:28.549 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 306, 2021-05-12T09:07:28.549 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:07:28.549 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:07:28.549 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 102, 2021-05-12T09:07:28.549 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:07:28.550 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:07:28.550 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 102, 2021-05-12T09:07:28.550 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:07:28.550 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 76, 2021-05-12T09:07:28.550 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 64, 2021-05-12T09:07:28.551 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 195, 2021-05-12T09:07:28.551 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 1210, 2021-05-12T09:07:28.551 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:07:28.551 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:07:28.552 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:07:28.552 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:07:28.552 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:07:28.552 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:07:28.552 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:07:28.553 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:07:28.553 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:07:28.553 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:07:28.553 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:07:28.553 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:07:28.554 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:07:28.554 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:07:28.554 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:07:28.554 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:07:28.554 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:07:28.555 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:07:28.555 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:07:28.555 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:07:28.555 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:07:28.556 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:07:28.556 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:07:28.556 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:07:28.556 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:07:28.557 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.557 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:07:28.557 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:07:28.557 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:07:28.557 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:07:28.558 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 0, 2021-05-12T09:07:28.558 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 0, 2021-05-12T09:07:28.558 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:07:28.558 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:07:28.558 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:07:28.559 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:07:28.559 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:07:28.559 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.559 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 30, 2021-05-12T09:07:28.559 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 30, 2021-05-12T09:07:28.560 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 4, 2021-05-12T09:07:28.560 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 4, 2021-05-12T09:07:28.560 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 0 2021-05-12T09:07:28.560 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.561 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_sum": { 2021-05-12T09:07:28.561 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 0, 2021-05-12T09:07:28.561 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 0, 2021-05-12T09:07:28.561 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pgs": 6, 2021-05-12T09:07:28.561 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osds": 3, 2021-05-12T09:07:28.562 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_osds": 3, 2021-05-12T09:07:28.562 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T09:07:28.562 INFO:tasks.workunit.client.0.smithi028.stdout: "kb": 314572800, 2021-05-12T09:07:28.562 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used": 21460, 2021-05-12T09:07:28.562 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_data": 3572, 2021-05-12T09:07:28.563 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_omap": 2, 2021-05-12T09:07:28.563 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_meta": 17853, 2021-05-12T09:07:28.563 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_avail": 314551340, 2021-05-12T09:07:28.563 INFO:tasks.workunit.client.0.smithi028.stdout: "statfs": { 2021-05-12T09:07:28.563 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 322122547200, 2021-05-12T09:07:28.564 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 322100572160, 2021-05-12T09:07:28.564 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:07:28.564 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 3657728, 2021-05-12T09:07:28.564 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1864857, 2021-05-12T09:07:28.564 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:07:28.565 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:07:28.565 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:07:28.565 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 2160, 2021-05-12T09:07:28.565 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 18282384 2021-05-12T09:07:28.565 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.566 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_peers": [], 2021-05-12T09:07:28.566 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_trim_queue_len": 0, 2021-05-12T09:07:28.566 INFO:tasks.workunit.client.0.smithi028.stdout: "num_snap_trimming": 0, 2021-05-12T09:07:28.566 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shards_repaired": 0, 2021-05-12T09:07:28.567 INFO:tasks.workunit.client.0.smithi028.stdout: "op_queue_age_hist": { 2021-05-12T09:07:28.567 INFO:tasks.workunit.client.0.smithi028.stdout: "histogram": [], 2021-05-12T09:07:28.567 INFO:tasks.workunit.client.0.smithi028.stdout: "upper_bound": 1 2021-05-12T09:07:28.567 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.567 INFO:tasks.workunit.client.0.smithi028.stdout: "perf_stat": { 2021-05-12T09:07:28.568 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ms": 0, 2021-05-12T09:07:28.568 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ms": 0, 2021-05-12T09:07:28.568 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ns": 0, 2021-05-12T09:07:28.568 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ns": 0 2021-05-12T09:07:28.568 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.569 INFO:tasks.workunit.client.0.smithi028.stdout: "alerts": [], 2021-05-12T09:07:28.569 INFO:tasks.workunit.client.0.smithi028.stdout: "network_ping_times": [] 2021-05-12T09:07:28.569 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.569 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_sum_by_class": { 2021-05-12T09:07:28.569 INFO:tasks.workunit.client.0.smithi028.stdout: "hdd": { 2021-05-12T09:07:28.570 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 0, 2021-05-12T09:07:28.570 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 0, 2021-05-12T09:07:28.570 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pgs": 6, 2021-05-12T09:07:28.570 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osds": 3, 2021-05-12T09:07:28.570 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_osds": 3, 2021-05-12T09:07:28.571 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T09:07:28.571 INFO:tasks.workunit.client.0.smithi028.stdout: "kb": 314572800, 2021-05-12T09:07:28.571 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used": 21460, 2021-05-12T09:07:28.571 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_data": 3572, 2021-05-12T09:07:28.572 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_omap": 2, 2021-05-12T09:07:28.572 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_meta": 17853, 2021-05-12T09:07:28.572 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_avail": 314551340, 2021-05-12T09:07:28.572 INFO:tasks.workunit.client.0.smithi028.stdout: "statfs": { 2021-05-12T09:07:28.572 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 322122547200, 2021-05-12T09:07:28.573 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 322100572160, 2021-05-12T09:07:28.573 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:07:28.573 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 3657728, 2021-05-12T09:07:28.573 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1864857, 2021-05-12T09:07:28.573 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:07:28.574 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:07:28.574 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:07:28.574 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 2160, 2021-05-12T09:07:28.574 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 18282384 2021-05-12T09:07:28.574 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.575 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_peers": [], 2021-05-12T09:07:28.575 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_trim_queue_len": 0, 2021-05-12T09:07:28.575 INFO:tasks.workunit.client.0.smithi028.stdout: "num_snap_trimming": 0, 2021-05-12T09:07:28.575 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shards_repaired": 0, 2021-05-12T09:07:28.575 INFO:tasks.workunit.client.0.smithi028.stdout: "op_queue_age_hist": { 2021-05-12T09:07:28.576 INFO:tasks.workunit.client.0.smithi028.stdout: "histogram": [], 2021-05-12T09:07:28.576 INFO:tasks.workunit.client.0.smithi028.stdout: "upper_bound": 1 2021-05-12T09:07:28.576 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.576 INFO:tasks.workunit.client.0.smithi028.stdout: "perf_stat": { 2021-05-12T09:07:28.576 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ms": 0, 2021-05-12T09:07:28.577 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ms": 0, 2021-05-12T09:07:28.577 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ns": 0, 2021-05-12T09:07:28.577 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ns": 0 2021-05-12T09:07:28.577 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.577 INFO:tasks.workunit.client.0.smithi028.stdout: "alerts": [], 2021-05-12T09:07:28.578 INFO:tasks.workunit.client.0.smithi028.stdout: "network_ping_times": [] 2021-05-12T09:07:28.578 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.578 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.578 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_stats": [ 2021-05-12T09:07:28.579 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.579 INFO:tasks.workunit.client.0.smithi028.stdout: "poolid": 2, 2021-05-12T09:07:28.579 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 1, 2021-05-12T09:07:28.579 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:07:28.579 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 48300, 2021-05-12T09:07:28.580 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 100, 2021-05-12T09:07:28.580 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:07:28.580 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 300, 2021-05-12T09:07:28.580 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:07:28.580 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:07:28.581 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 100, 2021-05-12T09:07:28.581 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:07:28.581 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:07:28.581 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 100, 2021-05-12T09:07:28.581 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:07:28.582 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 0, 2021-05-12T09:07:28.582 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 0, 2021-05-12T09:07:28.582 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 100, 2021-05-12T09:07:28.582 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 100, 2021-05-12T09:07:28.582 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:07:28.583 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:07:28.583 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:07:28.583 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:07:28.583 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:07:28.583 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:07:28.584 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:07:28.584 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:07:28.584 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:07:28.584 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:07:28.585 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:07:28.585 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:07:28.585 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:07:28.585 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:07:28.585 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:07:28.586 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:07:28.586 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:07:28.586 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:07:28.586 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:07:28.586 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:07:28.587 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:07:28.587 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:07:28.587 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:07:28.587 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:07:28.587 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:07:28.588 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.588 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:07:28.588 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:07:28.588 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:07:28.589 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:07:28.589 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 1220608, 2021-05-12T09:07:28.589 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 143228, 2021-05-12T09:07:28.589 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:07:28.589 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:07:28.590 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:07:28.590 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:07:28.590 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:07:28.590 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.590 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 16, 2021-05-12T09:07:28.591 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 16, 2021-05-12T09:07:28.591 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 2, 2021-05-12T09:07:28.591 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 2, 2021-05-12T09:07:28.591 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 3 2021-05-12T09:07:28.591 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.592 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.592 INFO:tasks.workunit.client.0.smithi028.stdout: "poolid": 1, 2021-05-12T09:07:28.592 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 1, 2021-05-12T09:07:28.592 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:07:28.593 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 459280, 2021-05-12T09:07:28.593 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 2, 2021-05-12T09:07:28.593 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:07:28.593 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 6, 2021-05-12T09:07:28.593 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:07:28.594 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:07:28.594 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 2, 2021-05-12T09:07:28.594 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:07:28.594 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:07:28.594 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 2, 2021-05-12T09:07:28.595 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:07:28.595 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 76, 2021-05-12T09:07:28.595 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 64, 2021-05-12T09:07:28.595 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 95, 2021-05-12T09:07:28.595 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 1110, 2021-05-12T09:07:28.596 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:07:28.596 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:07:28.596 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:07:28.596 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:07:28.597 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:07:28.597 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:07:28.597 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:07:28.597 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:07:28.597 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:07:28.598 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:07:28.598 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:07:28.598 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:07:28.598 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:07:28.598 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:07:28.599 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:07:28.599 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:07:28.599 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:07:28.599 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:07:28.599 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:07:28.600 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:07:28.600 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:07:28.600 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:07:28.600 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:07:28.601 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:07:28.601 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:07:28.601 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.601 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:07:28.601 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:07:28.602 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:07:28.602 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:07:28.602 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 1388544, 2021-05-12T09:07:28.602 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1377840, 2021-05-12T09:07:28.602 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:07:28.603 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:07:28.603 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:07:28.603 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:07:28.603 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:07:28.603 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.604 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 14, 2021-05-12T09:07:28.604 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 14, 2021-05-12T09:07:28.604 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 2, 2021-05-12T09:07:28.604 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 2, 2021-05-12T09:07:28.604 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 3 2021-05-12T09:07:28.605 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.605 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.605 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_stats": [ 2021-05-12T09:07:28.605 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.605 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:07:28.606 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 167503724555 2021-05-12T09:07:28.606 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.606 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.606 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:07:28.607 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 176093659146 2021-05-12T09:07:28.607 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.607 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.607 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:07:28.607 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 64424509455 2021-05-12T09:07:28.608 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.608 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.608 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_by_state": [ 2021-05-12T09:07:28.608 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.608 INFO:tasks.workunit.client.0.smithi028.stdout: "state": "active+undersized+degraded", 2021-05-12T09:07:28.609 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 2 2021-05-12T09:07:28.609 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.609 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.609 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_by_osd": [ 2021-05-12T09:07:28.609 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.610 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:07:28.610 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:07:28.610 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 0, 2021-05-12T09:07:28.610 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:07:28.610 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.611 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.611 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:07:28.611 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 2, 2021-05-12T09:07:28.611 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 2, 2021-05-12T09:07:28.611 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:07:28.612 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.612 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.612 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:07:28.612 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:07:28.612 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 2, 2021-05-12T09:07:28.613 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:07:28.613 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.613 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.613 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": [ 2021-05-12T09:07:28.614 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.614 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 1, 2021-05-12T09:07:28.614 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": {} 2021-05-12T09:07:28.614 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.614 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:07:28.615 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 2, 2021-05-12T09:07:28.615 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": {} 2021-05-12T09:07:28.615 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.615 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:07:28.615 INFO:tasks.workunit.client.0.smithi028.stdout: "servicemap": { 2021-05-12T09:07:28.616 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 5, 2021-05-12T09:07:28.616 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:06:57.819851+0000", 2021-05-12T09:07:28.616 INFO:tasks.workunit.client.0.smithi028.stdout: "services": { 2021-05-12T09:07:28.616 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": { 2021-05-12T09:07:28.616 INFO:tasks.workunit.client.0.smithi028.stdout: "daemons": { 2021-05-12T09:07:28.617 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": "", 2021-05-12T09:07:28.617 INFO:tasks.workunit.client.0.smithi028.stdout: "0": { 2021-05-12T09:07:28.617 INFO:tasks.workunit.client.0.smithi028.stdout: "start_epoch": 0, 2021-05-12T09:07:28.617 INFO:tasks.workunit.client.0.smithi028.stdout: "start_stamp": "0.000000", 2021-05-12T09:07:28.617 INFO:tasks.workunit.client.0.smithi028.stdout: "gid": 0, 2021-05-12T09:07:28.618 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T09:07:28.618 INFO:tasks.workunit.client.0.smithi028.stdout: "metadata": {}, 2021-05-12T09:07:28.618 INFO:tasks.workunit.client.0.smithi028.stdout: "task_status": {} 2021-05-12T09:07:28.618 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.618 INFO:tasks.workunit.client.0.smithi028.stdout: "1": { 2021-05-12T09:07:28.619 INFO:tasks.workunit.client.0.smithi028.stdout: "start_epoch": 0, 2021-05-12T09:07:28.619 INFO:tasks.workunit.client.0.smithi028.stdout: "start_stamp": "0.000000", 2021-05-12T09:07:28.619 INFO:tasks.workunit.client.0.smithi028.stdout: "gid": 0, 2021-05-12T09:07:28.619 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T09:07:28.620 INFO:tasks.workunit.client.0.smithi028.stdout: "metadata": {}, 2021-05-12T09:07:28.620 INFO:tasks.workunit.client.0.smithi028.stdout: "task_status": {} 2021-05-12T09:07:28.620 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.620 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.620 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.621 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.621 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:07:28.621 INFO:tasks.workunit.client.0.smithi028.stdout: "paxos": { 2021-05-12T09:07:28.621 INFO:tasks.workunit.client.0.smithi028.stdout: "first_committed": 1, 2021-05-12T09:07:28.621 INFO:tasks.workunit.client.0.smithi028.stdout: "last_committed": 374, 2021-05-12T09:07:28.622 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pn": 0, 2021-05-12T09:07:28.622 INFO:tasks.workunit.client.0.smithi028.stdout: "accepted_pn": 0 2021-05-12T09:07:28.622 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:07:28.622 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:07:28.623 INFO:tasks.workunit.client.0.smithi028.stderr:report 4143996360 2021-05-12T09:07:28.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T09:07:28.624 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2021-05-12T09:07:28.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2021-05-12T09:07:28.624 INFO:tasks.workunit.client.0.smithi028.stdout:writing non-divergent object existing_4 2021-05-12T09:07:28.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2021-05-12T09:07:28.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2021-05-12T09:07:28.704 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:07:28.705 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T09:06:53.004909+0000 32'100 47:121 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:05:59.322736+0000 0'0 2021-05-12T09:05:59.322736+0000 0 2021-05-12T09:07:28.705 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:06:52.929459+0000 32'62 47:115 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:05:55.908046+0000 0'0 2021-05-12T09:05:55.908046+0000 0 2021-05-12T09:07:28.705 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:07:28.706 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:07:28.707 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:07:28.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2021-05-12T09:07:28.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T09:07:28.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2021-05-12T09:07:28.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:07:28.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:07:28.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:07:28.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:07:28.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T09:07:28.835 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:07:28.835 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:07:28.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:07:28.836 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:07:28.836 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:07:28.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:07:28.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:07:28.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:07:28.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:07:28.893 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:07:29.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:07:29.282 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:07:29.282 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:07:29.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:07:29.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:07:29.283 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:07:29.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=167503724557 2021-05-12T09:07:29.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 167503724557 2021-05-12T09:07:29.424 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-167503724557' 2021-05-12T09:07:29.424 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:07:29.425 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:07:29.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=176093659148 2021-05-12T09:07:29.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 176093659148 2021-05-12T09:07:29.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-167503724557 1-176093659148' 2021-05-12T09:07:29.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:07:29.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:07:29.665 INFO:tasks.workunit.client.0.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-05-12T09:07:29.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T09:07:29.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T09:07:29.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T09:07:29.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:07:29.677 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-167503724557 2021-05-12T09:07:29.678 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:07:29.678 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:07:29.679 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-167503724557 2021-05-12T09:07:29.679 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:07:29.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=167503724557 2021-05-12T09:07:29.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 167503724557' 2021-05-12T09:07:29.682 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 167503724557 2021-05-12T09:07:29.683 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:07:30.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 167503724557 -lt 167503724557 2021-05-12T09:07:30.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:07:30.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-176093659148 2021-05-12T09:07:30.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:07:30.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:07:30.063 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-176093659148 2021-05-12T09:07:30.063 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:07:30.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=176093659148 2021-05-12T09:07:30.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 176093659148' 2021-05-12T09:07:30.064 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 176093659148 2021-05-12T09:07:30.065 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:07:30.447 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 176093659148 -lt 176093659148 2021-05-12T09:07:30.448 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:07:30.448 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:30.449 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:30.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:07:30.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:07:30.954 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:07:30.955 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:07:30.955 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:07:30.956 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:07:30.956 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:07:30.956 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:07:31.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:07:31.328 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:07:31.328 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:31.329 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:31.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:07:31.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T09:07:31.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T09:07:31.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T09:07:31.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:07:31.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:07:31.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:07:31.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:07:31.938 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:07:31.938 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:07:31.939 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:07:31.939 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:07:31.939 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:07:31.939 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:07:32.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:07:32.319 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:07:32.319 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:32.319 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:32.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:07:32.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:07:32.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:07:32.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:07:32.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:07:32.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:07:32.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:07:32.823 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:07:32.823 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:07:33.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:07:33.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:07:33.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T09:07:33.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:07:33.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T09:07:33.528 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:07:33.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:07:33.529 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:07:33.529 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:07:33.530 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:07:33.530 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:07:33.530 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:07:33.530 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:07:33.908 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:07:33.909 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:07:33.909 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:33.910 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:34.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:07:34.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:07:34.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:07:34.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:07:34.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:07:34.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:07:34.433 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:07:34.433 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:07:34.433 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:07:34.941 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:07:34.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:07:34.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T09:07:34.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:07:34.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T09:07:35.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:07:35.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:07:35.344 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:07:35.345 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:07:35.345 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:07:35.345 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:07:35.345 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:07:35.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:07:35.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:07:35.724 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:07:35.724 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:35.725 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:36.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:07:36.238 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:07:36.238 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:07:36.238 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:07:36.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:07:36.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:07:36.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:07:36.239 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:07:36.239 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:07:36.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:07:36.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:07:36.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T09:07:36.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:07:36.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T09:07:37.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:07:37.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:07:37.546 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:07:37.547 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:07:37.547 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:07:37.547 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:07:37.547 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:07:37.548 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:07:37.937 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:07:37.937 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:07:37.937 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:37.938 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:38.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:07:38.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:07:38.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:07:38.458 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:07:38.458 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:07:38.458 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:07:38.458 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:07:38.458 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:07:38.459 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:07:38.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:07:38.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:07:38.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T09:07:38.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:07:38.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T09:07:40.569 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:07:40.569 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:07:40.570 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:07:40.570 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:07:40.570 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:07:40.570 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:07:40.571 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:07:40.571 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:07:40.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:07:40.959 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:07:40.959 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:40.960 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:41.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:07:41.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:07:41.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:07:41.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:07:41.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:07:41.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:07:41.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:07:41.471 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:07:41.471 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:07:41.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:07:41.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:07:41.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T09:07:41.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:07:41.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T09:07:45.179 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:07:45.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:07:45.180 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:07:45.180 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:07:45.180 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:07:45.181 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:07:45.181 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:07:45.181 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:07:45.568 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:07:45.569 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:07:45.569 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:45.570 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:46.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:07:46.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:07:46.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:07:46.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:07:46.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:07:46.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:07:46.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:07:46.091 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:07:46.091 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:07:46.595 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:07:46.595 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:07:46.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T09:07:46.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:07:46.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T09:07:52.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:07:52.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:07:52.997 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:07:52.998 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:07:52.998 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:07:52.998 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:07:52.998 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:07:52.999 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:07:53.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:07:53.387 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:07:53.387 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:07:53.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:07:53.902 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:07:53.902 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:07:53.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:07:53.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:07:53.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:07:53.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:07:53.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:07:53.904 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:07:53.904 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:07:54.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:07:54.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:07:54.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T09:07:54.410 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:07:54.410 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T09:08:01.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:08:01.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:08:01.711 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:08:01.711 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:08:01.711 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:08:01.712 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:08:01.712 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:08:01.712 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:08:02.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:08:02.098 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:08:02.099 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:08:02.100 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:08:02.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:08:02.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:08:02.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:08:02.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:08:02.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:08:02.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:08:02.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:08:02.610 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:08:02.610 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:08:03.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:08:03.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:08:03.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T09:08:03.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T09:08:03.611 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:08:03.611 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_fingerprint": "b157a664-9ad2-403b-83e9-0841dc9dbc16", 2021-05-12T09:08:03.611 INFO:tasks.workunit.client.0.smithi028.stdout: "version": "17.0.0-4095-ga8c673d5", 2021-05-12T09:08:03.611 INFO:tasks.workunit.client.0.smithi028.stdout: "commit": "a8c673d555488beec9ccda5812ba822e69c12ea9", 2021-05-12T09:08:03.612 INFO:tasks.workunit.client.0.smithi028.stdout: "timestamp": "2021-05-12T09:08:03.610033+0000", 2021-05-12T09:08:03.612 INFO:tasks.workunit.client.0.smithi028.stdout: "tag": "", 2021-05-12T09:08:03.613 INFO:tasks.workunit.client.0.smithi028.stdout: "health": { 2021-05-12T09:08:03.613 INFO:tasks.workunit.client.0.smithi028.stdout: "status": "HEALTH_WARN", 2021-05-12T09:08:03.613 INFO:tasks.workunit.client.0.smithi028.stdout: "checks": { 2021-05-12T09:08:03.613 INFO:tasks.workunit.client.0.smithi028.stdout: "OSDMAP_FLAGS": { 2021-05-12T09:08:03.614 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:08:03.614 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:08:03.614 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T09:08:03.614 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 17 2021-05-12T09:08:03.615 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.615 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [], 2021-05-12T09:08:03.615 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:08:03.615 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.615 INFO:tasks.workunit.client.0.smithi028.stdout: "OSD_DOWN": { 2021-05-12T09:08:03.616 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:08:03.616 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:08:03.616 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "1 osds down", 2021-05-12T09:08:03.616 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 1 2021-05-12T09:08:03.617 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.617 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:08:03.617 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.617 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "osd.2 (root=default,host=smithi028) is down" 2021-05-12T09:08:03.617 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.618 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.618 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:08:03.618 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.618 INFO:tasks.workunit.client.0.smithi028.stdout: "PG_DEGRADED": { 2021-05-12T09:08:03.619 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:08:03.619 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:08:03.619 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "Degraded data redundancy: 102/306 objects degraded (33.333%), 2 pgs degraded, 2 pgs undersized", 2021-05-12T09:08:03.619 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 4 2021-05-12T09:08:03.620 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.620 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:08:03.620 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.620 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 1.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:08:03.620 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.621 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.621 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:08:03.621 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.622 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.622 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:08:03.622 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.622 INFO:tasks.workunit.client.0.smithi028.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T09:08:03.622 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:08:03.623 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:08:03.623 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T09:08:03.623 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 1 2021-05-12T09:08:03.623 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.624 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:08:03.624 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.624 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T09:08:03.624 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.624 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.625 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T09:08:03.625 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.625 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.625 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:08:03.626 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.626 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.626 INFO:tasks.workunit.client.0.smithi028.stdout: "mutes": [] 2021-05-12T09:08:03.626 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.626 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap_first_committed": 1, 2021-05-12T09:08:03.627 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap_last_committed": 1, 2021-05-12T09:08:03.627 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap": { 2021-05-12T09:08:03.627 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 1, 2021-05-12T09:08:03.628 INFO:tasks.workunit.client.0.smithi028.stdout: "fsid": "8342ec9e-531f-44b4-8c8c-079d667fe908", 2021-05-12T09:08:03.628 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:05:27.779860Z", 2021-05-12T09:08:03.628 INFO:tasks.workunit.client.0.smithi028.stdout: "created": "2021-05-12T09:05:27.779860Z", 2021-05-12T09:08:03.629 INFO:tasks.workunit.client.0.smithi028.stdout: "min_mon_release": 17, 2021-05-12T09:08:03.629 INFO:tasks.workunit.client.0.smithi028.stdout: "min_mon_release_name": "quincy", 2021-05-12T09:08:03.629 INFO:tasks.workunit.client.0.smithi028.stdout: "election_strategy": 1, 2021-05-12T09:08:03.629 INFO:tasks.workunit.client.0.smithi028.stdout: "disallowed_leaders: ": "", 2021-05-12T09:08:03.629 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode": false, 2021-05-12T09:08:03.630 INFO:tasks.workunit.client.0.smithi028.stdout: "features": { 2021-05-12T09:08:03.630 INFO:tasks.workunit.client.0.smithi028.stdout: "persistent": [ 2021-05-12T09:08:03.630 INFO:tasks.workunit.client.0.smithi028.stdout: "kraken", 2021-05-12T09:08:03.630 INFO:tasks.workunit.client.0.smithi028.stdout: "luminous", 2021-05-12T09:08:03.631 INFO:tasks.workunit.client.0.smithi028.stdout: "mimic", 2021-05-12T09:08:03.631 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap-prune", 2021-05-12T09:08:03.631 INFO:tasks.workunit.client.0.smithi028.stdout: "nautilus", 2021-05-12T09:08:03.631 INFO:tasks.workunit.client.0.smithi028.stdout: "octopus", 2021-05-12T09:08:03.632 INFO:tasks.workunit.client.0.smithi028.stdout: "pacific", 2021-05-12T09:08:03.632 INFO:tasks.workunit.client.0.smithi028.stdout: "elector-pinging", 2021-05-12T09:08:03.632 INFO:tasks.workunit.client.0.smithi028.stdout: "quincy" 2021-05-12T09:08:03.632 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.633 INFO:tasks.workunit.client.0.smithi028.stdout: "optional": [] 2021-05-12T09:08:03.633 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.633 INFO:tasks.workunit.client.0.smithi028.stdout: "mons": [ 2021-05-12T09:08:03.633 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.634 INFO:tasks.workunit.client.0.smithi028.stdout: "rank": 0, 2021-05-12T09:08:03.634 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "a", 2021-05-12T09:08:03.634 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:08:03.634 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:08:03.634 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.635 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:08:03.635 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:7115", 2021-05-12T09:08:03.635 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 0 2021-05-12T09:08:03.635 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.636 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.636 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.636 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T09:08:03.636 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T09:08:03.636 INFO:tasks.workunit.client.0.smithi028.stdout: "priority": 0, 2021-05-12T09:08:03.637 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 0, 2021-05-12T09:08:03.637 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_location": "{}" 2021-05-12T09:08:03.637 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.637 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.638 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.638 INFO:tasks.workunit.client.0.smithi028.stdout: "quorum": [ 2021-05-12T09:08:03.638 INFO:tasks.workunit.client.0.smithi028.stdout: 0 2021-05-12T09:08:03.638 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.639 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap": { 2021-05-12T09:08:03.639 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 47, 2021-05-12T09:08:03.639 INFO:tasks.workunit.client.0.smithi028.stdout: "fsid": "8342ec9e-531f-44b4-8c8c-079d667fe908", 2021-05-12T09:08:03.639 INFO:tasks.workunit.client.0.smithi028.stdout: "created": "2021-05-12T09:05:28.578482+0000", 2021-05-12T09:08:03.640 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:06:52.602275+0000", 2021-05-12T09:08:03.640 INFO:tasks.workunit.client.0.smithi028.stdout: "last_up_change": "2021-05-12T09:06:50.813501+0000", 2021-05-12T09:08:03.640 INFO:tasks.workunit.client.0.smithi028.stdout: "last_in_change": "2021-05-12T09:06:51.889810+0000", 2021-05-12T09:08:03.640 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T09:08:03.641 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_num": 5800384, 2021-05-12T09:08:03.641 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_set": [ 2021-05-12T09:08:03.641 INFO:tasks.workunit.client.0.smithi028.stdout: "nodown", 2021-05-12T09:08:03.641 INFO:tasks.workunit.client.0.smithi028.stdout: "noin", 2021-05-12T09:08:03.642 INFO:tasks.workunit.client.0.smithi028.stdout: "noout", 2021-05-12T09:08:03.642 INFO:tasks.workunit.client.0.smithi028.stdout: "pglog_hardlimit", 2021-05-12T09:08:03.642 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snapdirs", 2021-05-12T09:08:03.642 INFO:tasks.workunit.client.0.smithi028.stdout: "recovery_deletes", 2021-05-12T09:08:03.642 INFO:tasks.workunit.client.0.smithi028.stdout: "sortbitwise" 2021-05-12T09:08:03.643 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.643 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_version": 9, 2021-05-12T09:08:03.643 INFO:tasks.workunit.client.0.smithi028.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T09:08:03.643 INFO:tasks.workunit.client.0.smithi028.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T09:08:03.643 INFO:tasks.workunit.client.0.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T09:08:03.644 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_snapshot": "", 2021-05-12T09:08:03.644 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_max": 2, 2021-05-12T09:08:03.644 INFO:tasks.workunit.client.0.smithi028.stdout: "max_osd": 3, 2021-05-12T09:08:03.644 INFO:tasks.workunit.client.0.smithi028.stdout: "require_min_compat_client": "luminous", 2021-05-12T09:08:03.645 INFO:tasks.workunit.client.0.smithi028.stdout: "min_compat_client": "jewel", 2021-05-12T09:08:03.645 INFO:tasks.workunit.client.0.smithi028.stdout: "require_osd_release": "quincy", 2021-05-12T09:08:03.645 INFO:tasks.workunit.client.0.smithi028.stdout: "pools": [ 2021-05-12T09:08:03.645 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.645 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 1, 2021-05-12T09:08:03.646 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_name": ".mgr", 2021-05-12T09:08:03.646 INFO:tasks.workunit.client.0.smithi028.stdout: "create_time": "2021-05-12T09:05:55.843224+0000", 2021-05-12T09:08:03.646 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": 1, 2021-05-12T09:08:03.646 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_names": "hashpspool", 2021-05-12T09:08:03.646 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:08:03.647 INFO:tasks.workunit.client.0.smithi028.stdout: "size": 3, 2021-05-12T09:08:03.647 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 2, 2021-05-12T09:08:03.647 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_rule": 0, 2021-05-12T09:08:03.647 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_count": 0, 2021-05-12T09:08:03.648 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_target": 0, 2021-05-12T09:08:03.648 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T09:08:03.648 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T09:08:03.648 INFO:tasks.workunit.client.0.smithi028.stdout: "object_hash": 2, 2021-05-12T09:08:03.649 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_autoscale_mode": "off", 2021-05-12T09:08:03.649 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num": 1, 2021-05-12T09:08:03.649 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num": 1, 2021-05-12T09:08:03.650 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num_target": 1, 2021-05-12T09:08:03.650 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_target": 1, 2021-05-12T09:08:03.650 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_pending": 1, 2021-05-12T09:08:03.650 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pg_merge_meta": { 2021-05-12T09:08:03.650 INFO:tasks.workunit.client.0.smithi028.stdout: "source_pgid": "0.0", 2021-05-12T09:08:03.651 INFO:tasks.workunit.client.0.smithi028.stdout: "ready_epoch": 0, 2021-05-12T09:08:03.651 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_started": 0, 2021-05-12T09:08:03.651 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": 0, 2021-05-12T09:08:03.651 INFO:tasks.workunit.client.0.smithi028.stdout: "source_version": "0'0", 2021-05-12T09:08:03.651 INFO:tasks.workunit.client.0.smithi028.stdout: "target_version": "0'0" 2021-05-12T09:08:03.652 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.652 INFO:tasks.workunit.client.0.smithi028.stdout: "last_change": "21", 2021-05-12T09:08:03.652 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend": "0", 2021-05-12T09:08:03.652 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T09:08:03.652 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T09:08:03.653 INFO:tasks.workunit.client.0.smithi028.stdout: "auid": 0, 2021-05-12T09:08:03.653 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_mode": "selfmanaged", 2021-05-12T09:08:03.653 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_seq": 0, 2021-05-12T09:08:03.653 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_epoch": 0, 2021-05-12T09:08:03.654 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_snaps": [], 2021-05-12T09:08:03.654 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps": "[]", 2021-05-12T09:08:03.654 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_bytes": 0, 2021-05-12T09:08:03.654 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_objects": 0, 2021-05-12T09:08:03.654 INFO:tasks.workunit.client.0.smithi028.stdout: "tiers": [], 2021-05-12T09:08:03.655 INFO:tasks.workunit.client.0.smithi028.stdout: "tier_of": -1, 2021-05-12T09:08:03.655 INFO:tasks.workunit.client.0.smithi028.stdout: "read_tier": -1, 2021-05-12T09:08:03.655 INFO:tasks.workunit.client.0.smithi028.stdout: "write_tier": -1, 2021-05-12T09:08:03.655 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_mode": "none", 2021-05-12T09:08:03.656 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_bytes": 0, 2021-05-12T09:08:03.656 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_objects": 0, 2021-05-12T09:08:03.656 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T09:08:03.656 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T09:08:03.657 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T09:08:03.657 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_flush_age": 0, 2021-05-12T09:08:03.657 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_evict_age": 0, 2021-05-12T09:08:03.658 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profile": "", 2021-05-12T09:08:03.658 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_params": { 2021-05-12T09:08:03.658 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "none" 2021-05-12T09:08:03.658 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.659 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_period": 0, 2021-05-12T09:08:03.659 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_count": 0, 2021-05-12T09:08:03.659 INFO:tasks.workunit.client.0.smithi028.stdout: "use_gmt_hitset": true, 2021-05-12T09:08:03.659 INFO:tasks.workunit.client.0.smithi028.stdout: "min_read_recency_for_promote": 0, 2021-05-12T09:08:03.659 INFO:tasks.workunit.client.0.smithi028.stdout: "min_write_recency_for_promote": 0, 2021-05-12T09:08:03.660 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T09:08:03.660 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_search_last_n": 0, 2021-05-12T09:08:03.660 INFO:tasks.workunit.client.0.smithi028.stdout: "grade_table": [], 2021-05-12T09:08:03.660 INFO:tasks.workunit.client.0.smithi028.stdout: "stripe_width": 0, 2021-05-12T09:08:03.660 INFO:tasks.workunit.client.0.smithi028.stdout: "expected_num_objects": 0, 2021-05-12T09:08:03.661 INFO:tasks.workunit.client.0.smithi028.stdout: "fast_read": false, 2021-05-12T09:08:03.661 INFO:tasks.workunit.client.0.smithi028.stdout: "options": { 2021-05-12T09:08:03.661 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_min": 1 2021-05-12T09:08:03.661 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.661 INFO:tasks.workunit.client.0.smithi028.stdout: "application_metadata": { 2021-05-12T09:08:03.662 INFO:tasks.workunit.client.0.smithi028.stdout: "mgr": {} 2021-05-12T09:08:03.662 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.662 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.662 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.663 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 2, 2021-05-12T09:08:03.663 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_name": "test", 2021-05-12T09:08:03.663 INFO:tasks.workunit.client.0.smithi028.stdout: "create_time": "2021-05-12T09:05:59.272553+0000", 2021-05-12T09:08:03.663 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": 1, 2021-05-12T09:08:03.663 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_names": "hashpspool", 2021-05-12T09:08:03.664 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:08:03.664 INFO:tasks.workunit.client.0.smithi028.stdout: "size": 3, 2021-05-12T09:08:03.664 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 2, 2021-05-12T09:08:03.664 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_rule": 0, 2021-05-12T09:08:03.665 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_count": 0, 2021-05-12T09:08:03.665 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_target": 0, 2021-05-12T09:08:03.665 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T09:08:03.665 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T09:08:03.666 INFO:tasks.workunit.client.0.smithi028.stdout: "object_hash": 2, 2021-05-12T09:08:03.666 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_autoscale_mode": "off", 2021-05-12T09:08:03.666 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num": 1, 2021-05-12T09:08:03.667 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num": 1, 2021-05-12T09:08:03.667 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num_target": 1, 2021-05-12T09:08:03.667 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_target": 1, 2021-05-12T09:08:03.667 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_pending": 1, 2021-05-12T09:08:03.668 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pg_merge_meta": { 2021-05-12T09:08:03.668 INFO:tasks.workunit.client.0.smithi028.stdout: "source_pgid": "0.0", 2021-05-12T09:08:03.668 INFO:tasks.workunit.client.0.smithi028.stdout: "ready_epoch": 0, 2021-05-12T09:08:03.668 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_started": 0, 2021-05-12T09:08:03.669 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": 0, 2021-05-12T09:08:03.669 INFO:tasks.workunit.client.0.smithi028.stdout: "source_version": "0'0", 2021-05-12T09:08:03.669 INFO:tasks.workunit.client.0.smithi028.stdout: "target_version": "0'0" 2021-05-12T09:08:03.670 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.670 INFO:tasks.workunit.client.0.smithi028.stdout: "last_change": "32", 2021-05-12T09:08:03.670 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend": "0", 2021-05-12T09:08:03.670 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T09:08:03.670 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T09:08:03.671 INFO:tasks.workunit.client.0.smithi028.stdout: "auid": 0, 2021-05-12T09:08:03.671 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_mode": "selfmanaged", 2021-05-12T09:08:03.671 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_seq": 0, 2021-05-12T09:08:03.671 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_epoch": 0, 2021-05-12T09:08:03.671 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_snaps": [], 2021-05-12T09:08:03.672 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps": "[]", 2021-05-12T09:08:03.672 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_bytes": 0, 2021-05-12T09:08:03.672 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_objects": 0, 2021-05-12T09:08:03.672 INFO:tasks.workunit.client.0.smithi028.stdout: "tiers": [], 2021-05-12T09:08:03.672 INFO:tasks.workunit.client.0.smithi028.stdout: "tier_of": -1, 2021-05-12T09:08:03.673 INFO:tasks.workunit.client.0.smithi028.stdout: "read_tier": -1, 2021-05-12T09:08:03.673 INFO:tasks.workunit.client.0.smithi028.stdout: "write_tier": -1, 2021-05-12T09:08:03.673 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_mode": "none", 2021-05-12T09:08:03.673 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_bytes": 0, 2021-05-12T09:08:03.674 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_objects": 0, 2021-05-12T09:08:03.674 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T09:08:03.674 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T09:08:03.674 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T09:08:03.675 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_flush_age": 0, 2021-05-12T09:08:03.675 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_evict_age": 0, 2021-05-12T09:08:03.675 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profile": "", 2021-05-12T09:08:03.675 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_params": { 2021-05-12T09:08:03.675 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "none" 2021-05-12T09:08:03.676 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.676 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_period": 0, 2021-05-12T09:08:03.676 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_count": 0, 2021-05-12T09:08:03.677 INFO:tasks.workunit.client.0.smithi028.stdout: "use_gmt_hitset": true, 2021-05-12T09:08:03.677 INFO:tasks.workunit.client.0.smithi028.stdout: "min_read_recency_for_promote": 0, 2021-05-12T09:08:03.677 INFO:tasks.workunit.client.0.smithi028.stdout: "min_write_recency_for_promote": 0, 2021-05-12T09:08:03.677 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T09:08:03.678 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_search_last_n": 0, 2021-05-12T09:08:03.678 INFO:tasks.workunit.client.0.smithi028.stdout: "grade_table": [], 2021-05-12T09:08:03.678 INFO:tasks.workunit.client.0.smithi028.stdout: "stripe_width": 0, 2021-05-12T09:08:03.678 INFO:tasks.workunit.client.0.smithi028.stdout: "expected_num_objects": 0, 2021-05-12T09:08:03.679 INFO:tasks.workunit.client.0.smithi028.stdout: "fast_read": false, 2021-05-12T09:08:03.679 INFO:tasks.workunit.client.0.smithi028.stdout: "options": {}, 2021-05-12T09:08:03.679 INFO:tasks.workunit.client.0.smithi028.stdout: "application_metadata": {} 2021-05-12T09:08:03.679 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.679 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.680 INFO:tasks.workunit.client.0.smithi028.stdout: "osds": [ 2021-05-12T09:08:03.680 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.680 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:08:03.680 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "c5a51d7a-897f-41a2-97c3-e9ef91d860e1", 2021-05-12T09:08:03.681 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 1, 2021-05-12T09:08:03.681 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:08:03.681 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:08:03.681 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:08:03.682 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 5, 2021-05-12T09:08:03.682 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 32, 2021-05-12T09:08:03.682 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 39, 2021-05-12T09:08:03.682 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 45, 2021-05-12T09:08:03.683 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 33, 2021-05-12T09:08:03.683 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:08:03.683 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:08:03.683 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:08:03.683 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.684 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:08:03.684 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6802", 2021-05-12T09:08:03.684 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:08:03.684 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.685 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.685 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:08:03.685 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6803", 2021-05-12T09:08:03.685 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:08:03.685 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.686 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.686 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.686 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:08:03.686 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:08:03.686 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.687 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:08:03.687 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6804", 2021-05-12T09:08:03.687 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:08:03.688 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.688 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.688 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:08:03.688 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6805", 2021-05-12T09:08:03.689 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:08:03.689 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.689 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.689 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.690 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:08:03.690 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:08:03.690 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.690 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:08:03.691 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6808", 2021-05-12T09:08:03.691 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:08:03.691 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.691 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.691 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:08:03.692 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6809", 2021-05-12T09:08:03.692 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:08:03.692 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.692 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.692 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.693 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:08:03.693 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:08:03.693 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.693 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:08:03.694 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6806", 2021-05-12T09:08:03.694 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:08:03.694 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.694 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.695 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:08:03.695 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6807", 2021-05-12T09:08:03.695 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49441 2021-05-12T09:08:03.695 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.695 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.696 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.696 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6803/49441", 2021-05-12T09:08:03.696 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6805/49441", 2021-05-12T09:08:03.696 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6809/49441", 2021-05-12T09:08:03.697 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6807/49441", 2021-05-12T09:08:03.697 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:08:03.697 INFO:tasks.workunit.client.0.smithi028.stdout: "exists", 2021-05-12T09:08:03.698 INFO:tasks.workunit.client.0.smithi028.stdout: "up" 2021-05-12T09:08:03.698 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.698 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.698 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.699 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:08:03.699 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "e003828b-f9e6-48de-bd5a-128e4af3e5dc", 2021-05-12T09:08:03.699 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 1, 2021-05-12T09:08:03.699 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:08:03.700 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:08:03.700 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:08:03.700 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 10, 2021-05-12T09:08:03.700 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 32, 2021-05-12T09:08:03.701 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 41, 2021-05-12T09:08:03.701 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 46, 2021-05-12T09:08:03.701 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 34, 2021-05-12T09:08:03.701 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:08:03.702 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:08:03.702 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:08:03.702 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.702 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:08:03.702 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6810", 2021-05-12T09:08:03.703 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:08:03.703 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.703 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.703 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:08:03.704 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6811", 2021-05-12T09:08:03.704 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:08:03.704 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.704 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.704 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.705 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:08:03.705 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:08:03.705 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.706 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:08:03.706 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6812", 2021-05-12T09:08:03.706 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:08:03.706 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.707 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.707 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:08:03.707 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6813", 2021-05-12T09:08:03.707 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:08:03.708 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.708 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.708 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.708 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:08:03.708 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:08:03.709 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.709 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:08:03.709 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6816", 2021-05-12T09:08:03.709 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:08:03.709 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.710 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.710 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:08:03.710 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6817", 2021-05-12T09:08:03.710 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:08:03.711 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.711 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.711 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.711 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:08:03.712 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:08:03.712 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.712 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:08:03.712 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6814", 2021-05-12T09:08:03.712 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:08:03.713 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.713 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.713 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:08:03.713 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6815", 2021-05-12T09:08:03.714 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 49998 2021-05-12T09:08:03.714 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.714 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.714 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.714 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6811/49998", 2021-05-12T09:08:03.715 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6813/49998", 2021-05-12T09:08:03.715 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6817/49998", 2021-05-12T09:08:03.715 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6815/49998", 2021-05-12T09:08:03.715 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:08:03.716 INFO:tasks.workunit.client.0.smithi028.stdout: "exists", 2021-05-12T09:08:03.716 INFO:tasks.workunit.client.0.smithi028.stdout: "up" 2021-05-12T09:08:03.716 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.716 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.717 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.717 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:08:03.717 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "10e9d857-0855-44ba-8b8e-8d823ab9204a", 2021-05-12T09:08:03.717 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 0, 2021-05-12T09:08:03.717 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:08:03.718 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:08:03.718 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:08:03.718 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 0, 2021-05-12T09:08:03.718 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 0, 2021-05-12T09:08:03.719 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 15, 2021-05-12T09:08:03.719 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 26, 2021-05-12T09:08:03.719 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 35, 2021-05-12T09:08:03.719 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:08:03.720 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:08:03.720 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:08:03.720 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.721 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:08:03.721 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6818", 2021-05-12T09:08:03.721 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:08:03.721 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.721 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.722 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:08:03.722 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6819", 2021-05-12T09:08:03.722 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:08:03.722 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.723 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.723 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.723 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:08:03.723 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:08:03.723 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.724 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:08:03.724 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6820", 2021-05-12T09:08:03.724 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:08:03.724 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.724 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.725 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:08:03.725 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6821", 2021-05-12T09:08:03.725 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:08:03.725 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.725 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.726 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.726 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:08:03.726 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:08:03.726 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.727 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:08:03.727 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6824", 2021-05-12T09:08:03.727 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:08:03.727 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.728 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.728 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:08:03.728 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6825", 2021-05-12T09:08:03.728 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:08:03.729 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.729 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.729 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.729 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:08:03.730 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:08:03.730 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.730 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:08:03.730 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6822", 2021-05-12T09:08:03.731 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:08:03.731 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.731 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.731 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:08:03.731 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6823", 2021-05-12T09:08:03.732 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 45341 2021-05-12T09:08:03.732 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.732 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.732 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.732 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6819/45341", 2021-05-12T09:08:03.733 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6821/45341", 2021-05-12T09:08:03.733 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6825/45341", 2021-05-12T09:08:03.733 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6823/45341", 2021-05-12T09:08:03.733 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:08:03.733 INFO:tasks.workunit.client.0.smithi028.stdout: "exists" 2021-05-12T09:08:03.734 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.734 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.734 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.734 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_xinfo": [ 2021-05-12T09:08:03.735 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.735 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:08:03.735 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:06:40.977681+0000", 2021-05-12T09:08:03.735 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:08:03.735 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:08:03.736 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:08:03.736 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:08:03.736 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:05:37.831220+0000", 2021-05-12T09:08:03.736 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:08:03.736 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.737 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.737 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:08:03.738 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:06:41.506912+0000", 2021-05-12T09:08:03.738 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:08:03.738 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:08:03.738 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:08:03.739 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:08:03.739 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:05:46.839891+0000", 2021-05-12T09:08:03.739 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:08:03.739 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.740 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.740 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:08:03.740 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:06:42.039554+0000", 2021-05-12T09:08:03.740 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:08:03.740 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:08:03.741 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:08:03.741 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:08:03.741 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:05:56.142117+0000", 2021-05-12T09:08:03.741 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:08:03.741 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.742 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.742 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_upmap": [], 2021-05-12T09:08:03.742 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_upmap_items": [], 2021-05-12T09:08:03.742 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_temp": [], 2021-05-12T09:08:03.742 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_temp": [], 2021-05-12T09:08:03.743 INFO:tasks.workunit.client.0.smithi028.stdout: "blocklist": {}, 2021-05-12T09:08:03.743 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profiles": { 2021-05-12T09:08:03.743 INFO:tasks.workunit.client.0.smithi028.stdout: "default": { 2021-05-12T09:08:03.743 INFO:tasks.workunit.client.0.smithi028.stdout: "k": "2", 2021-05-12T09:08:03.743 INFO:tasks.workunit.client.0.smithi028.stdout: "m": "2", 2021-05-12T09:08:03.744 INFO:tasks.workunit.client.0.smithi028.stdout: "plugin": "jerasure", 2021-05-12T09:08:03.744 INFO:tasks.workunit.client.0.smithi028.stdout: "technique": "reed_sol_van" 2021-05-12T09:08:03.744 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.744 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.745 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps_queue": [], 2021-05-12T09:08:03.745 INFO:tasks.workunit.client.0.smithi028.stdout: "new_removed_snaps": [], 2021-05-12T09:08:03.745 INFO:tasks.workunit.client.0.smithi028.stdout: "new_purged_snaps": [], 2021-05-12T09:08:03.745 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_node_flags": {}, 2021-05-12T09:08:03.745 INFO:tasks.workunit.client.0.smithi028.stdout: "device_class_flags": {}, 2021-05-12T09:08:03.746 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode": { 2021-05-12T09:08:03.746 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode_enabled": false, 2021-05-12T09:08:03.747 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_bucket_count": 0, 2021-05-12T09:08:03.747 INFO:tasks.workunit.client.0.smithi028.stdout: "degraded_stretch_mode": 0, 2021-05-12T09:08:03.747 INFO:tasks.workunit.client.0.smithi028.stdout: "recovering_stretch_mode": 0, 2021-05-12T09:08:03.748 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode_bucket": 0 2021-05-12T09:08:03.748 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.748 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.748 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_metadata": [ 2021-05-12T09:08:03.748 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.749 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:08:03.749 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:08:03.749 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6804/49441,v1:127.0.0.1:6805/49441]", 2021-05-12T09:08:03.749 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:08:03.749 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:08:03.750 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:08:03.750 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:08:03.750 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:08:03.750 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:08:03.751 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:08:03.751 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:08:03.751 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:08:03.751 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T09:08:03.751 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:08:03.752 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:08:03.752 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:08:03.752 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:08:03.752 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:08:03.753 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:08:03.753 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:08:03.753 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:08:03.753 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:08:03.753 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:08:03.754 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:08:03.754 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:08:03.754 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:08:03.755 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:08:03.755 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:08:03.755 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6802/49441,v1:127.0.0.1:6803/49441]", 2021-05-12T09:08:03.755 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:08:03.755 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/49441,v1:127.0.0.1:6809/49441]", 2021-05-12T09:08:03.756 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/49441,v1:127.0.0.1:6807/49441]", 2021-05-12T09:08:03.756 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:08:03.756 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:08:03.756 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:08:03.757 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:08:03.757 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:08:03.757 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:08:03.758 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:08:03.758 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:08:03.758 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:08:03.758 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T09:08:03.759 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:08:03.759 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:08:03.759 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:08:03.759 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.759 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.760 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:08:03.760 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:08:03.760 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6812/49998,v1:127.0.0.1:6813/49998]", 2021-05-12T09:08:03.760 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:08:03.760 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:08:03.761 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:08:03.761 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:08:03.761 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:08:03.761 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:08:03.762 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:08:03.762 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:08:03.762 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:08:03.762 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T09:08:03.763 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:08:03.763 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:08:03.763 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:08:03.763 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:08:03.764 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:08:03.764 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:08:03.764 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:08:03.764 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:08:03.764 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:08:03.765 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:08:03.765 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:08:03.765 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:08:03.765 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:08:03.765 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:08:03.766 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:08:03.766 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6810/49998,v1:127.0.0.1:6811/49998]", 2021-05-12T09:08:03.766 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:08:03.766 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/49998,v1:127.0.0.1:6817/49998]", 2021-05-12T09:08:03.767 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/49998,v1:127.0.0.1:6815/49998]", 2021-05-12T09:08:03.767 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:08:03.767 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:08:03.767 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:08:03.768 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:08:03.768 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:08:03.768 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:08:03.768 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:08:03.768 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:08:03.769 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:08:03.769 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T09:08:03.769 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:08:03.769 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:08:03.770 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:08:03.770 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.770 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.771 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:08:03.771 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:08:03.771 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6820/45341,v1:127.0.0.1:6821/45341]", 2021-05-12T09:08:03.771 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:08:03.771 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:08:03.772 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:08:03.772 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:08:03.772 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:08:03.772 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:08:03.772 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:08:03.773 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:08:03.773 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:08:03.773 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T09:08:03.773 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:08:03.774 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:08:03.774 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:08:03.774 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:08:03.774 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:08:03.775 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:08:03.775 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:08:03.775 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:08:03.775 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:08:03.775 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:08:03.776 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:08:03.776 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:08:03.777 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:08:03.777 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:08:03.777 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:08:03.777 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6818/45341,v1:127.0.0.1:6819/45341]", 2021-05-12T09:08:03.778 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:08:03.778 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/45341,v1:127.0.0.1:6825/45341]", 2021-05-12T09:08:03.778 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/45341,v1:127.0.0.1:6823/45341]", 2021-05-12T09:08:03.778 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:08:03.779 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:08:03.779 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:08:03.779 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:08:03.779 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:08:03.779 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:08:03.780 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:08:03.780 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:08:03.780 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:08:03.780 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T09:08:03.781 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:08:03.781 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:08:03.781 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:08:03.781 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.781 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.782 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_clean_epochs": { 2021-05-12T09:08:03.782 INFO:tasks.workunit.client.0.smithi028.stdout: "min_last_epoch_clean": 0, 2021-05-12T09:08:03.782 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": { 2021-05-12T09:08:03.782 INFO:tasks.workunit.client.0.smithi028.stdout: "per_pool": [] 2021-05-12T09:08:03.783 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.783 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_epochs": [ 2021-05-12T09:08:03.783 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.783 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:08:03.783 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 39 2021-05-12T09:08:03.784 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.784 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.784 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:08:03.784 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 42 2021-05-12T09:08:03.785 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.785 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.785 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:08:03.785 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 15 2021-05-12T09:08:03.785 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.786 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.786 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.786 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_first_committed": 1, 2021-05-12T09:08:03.787 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_last_committed": 47, 2021-05-12T09:08:03.787 INFO:tasks.workunit.client.0.smithi028.stdout: "crushmap": { 2021-05-12T09:08:03.787 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": [ 2021-05-12T09:08:03.787 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.788 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:08:03.788 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.0", 2021-05-12T09:08:03.788 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:08:03.788 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.788 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.789 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:08:03.789 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.1", 2021-05-12T09:08:03.789 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:08:03.789 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.789 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.790 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:08:03.790 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.2", 2021-05-12T09:08:03.790 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:08:03.790 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.791 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.791 INFO:tasks.workunit.client.0.smithi028.stdout: "types": [ 2021-05-12T09:08:03.791 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.791 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 0, 2021-05-12T09:08:03.791 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd" 2021-05-12T09:08:03.792 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.792 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.792 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:08:03.792 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "host" 2021-05-12T09:08:03.792 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.793 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.793 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 2, 2021-05-12T09:08:03.793 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "chassis" 2021-05-12T09:08:03.794 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.794 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.794 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 3, 2021-05-12T09:08:03.794 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "rack" 2021-05-12T09:08:03.795 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.795 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.795 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 4, 2021-05-12T09:08:03.795 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "row" 2021-05-12T09:08:03.796 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.796 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.796 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 5, 2021-05-12T09:08:03.796 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "pdu" 2021-05-12T09:08:03.797 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.797 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.797 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 6, 2021-05-12T09:08:03.797 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "pod" 2021-05-12T09:08:03.798 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.798 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.798 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 7, 2021-05-12T09:08:03.798 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "room" 2021-05-12T09:08:03.798 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.799 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.799 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 8, 2021-05-12T09:08:03.799 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "datacenter" 2021-05-12T09:08:03.800 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.800 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.800 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 9, 2021-05-12T09:08:03.800 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "zone" 2021-05-12T09:08:03.801 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.801 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.801 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 10, 2021-05-12T09:08:03.801 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "region" 2021-05-12T09:08:03.802 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.802 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.802 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:08:03.802 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "root" 2021-05-12T09:08:03.803 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.803 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.803 INFO:tasks.workunit.client.0.smithi028.stdout: "buckets": [ 2021-05-12T09:08:03.803 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.804 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -1, 2021-05-12T09:08:03.804 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "default", 2021-05-12T09:08:03.804 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:08:03.804 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "root", 2021-05-12T09:08:03.804 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:08:03.805 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:08:03.805 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:08:03.805 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:08:03.805 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.805 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -3, 2021-05-12T09:08:03.806 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:08:03.806 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:08:03.806 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.806 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.807 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.807 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.807 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -2, 2021-05-12T09:08:03.807 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "default~hdd", 2021-05-12T09:08:03.808 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:08:03.808 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "root", 2021-05-12T09:08:03.808 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:08:03.808 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:08:03.808 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:08:03.809 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:08:03.809 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.809 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -4, 2021-05-12T09:08:03.809 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:08:03.810 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:08:03.810 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.810 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.810 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.811 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.811 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -3, 2021-05-12T09:08:03.811 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "smithi028", 2021-05-12T09:08:03.811 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:08:03.812 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "host", 2021-05-12T09:08:03.812 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:08:03.812 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:08:03.812 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:08:03.812 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:08:03.813 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.813 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:08:03.813 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:08:03.813 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:08:03.813 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.814 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.814 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:08:03.814 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:08:03.814 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 1 2021-05-12T09:08:03.815 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.815 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.815 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:08:03.815 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:08:03.815 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 2 2021-05-12T09:08:03.816 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.816 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.816 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.817 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.817 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -4, 2021-05-12T09:08:03.817 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "smithi028~hdd", 2021-05-12T09:08:03.817 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:08:03.818 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "host", 2021-05-12T09:08:03.818 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:08:03.818 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:08:03.818 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:08:03.819 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:08:03.819 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.819 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:08:03.819 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:08:03.819 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:08:03.820 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.820 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.820 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:08:03.820 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:08:03.821 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 1 2021-05-12T09:08:03.821 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.821 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.821 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:08:03.821 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:08:03.822 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 2 2021-05-12T09:08:03.822 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.822 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.822 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.822 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.823 INFO:tasks.workunit.client.0.smithi028.stdout: "rules": [ 2021-05-12T09:08:03.823 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.823 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_id": 0, 2021-05-12T09:08:03.823 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_name": "replicated_rule", 2021-05-12T09:08:03.824 INFO:tasks.workunit.client.0.smithi028.stdout: "ruleset": 0, 2021-05-12T09:08:03.824 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:08:03.824 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 1, 2021-05-12T09:08:03.824 INFO:tasks.workunit.client.0.smithi028.stdout: "max_size": 10, 2021-05-12T09:08:03.824 INFO:tasks.workunit.client.0.smithi028.stdout: "steps": [ 2021-05-12T09:08:03.825 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.825 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "take", 2021-05-12T09:08:03.825 INFO:tasks.workunit.client.0.smithi028.stdout: "item": -1, 2021-05-12T09:08:03.825 INFO:tasks.workunit.client.0.smithi028.stdout: "item_name": "default" 2021-05-12T09:08:03.825 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.826 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.826 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "choose_firstn", 2021-05-12T09:08:03.826 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 0, 2021-05-12T09:08:03.827 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "osd" 2021-05-12T09:08:03.827 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.827 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.827 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "emit" 2021-05-12T09:08:03.827 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.828 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:08:03.828 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.828 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.828 INFO:tasks.workunit.client.0.smithi028.stdout: "tunables": { 2021-05-12T09:08:03.829 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_local_tries": 0, 2021-05-12T09:08:03.829 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_local_fallback_tries": 0, 2021-05-12T09:08:03.829 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_total_tries": 50, 2021-05-12T09:08:03.829 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_descend_once": 1, 2021-05-12T09:08:03.830 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_vary_r": 1, 2021-05-12T09:08:03.830 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_stable": 1, 2021-05-12T09:08:03.830 INFO:tasks.workunit.client.0.smithi028.stdout: "straw_calc_version": 1, 2021-05-12T09:08:03.830 INFO:tasks.workunit.client.0.smithi028.stdout: "allowed_bucket_algs": 54, 2021-05-12T09:08:03.830 INFO:tasks.workunit.client.0.smithi028.stdout: "profile": "jewel", 2021-05-12T09:08:03.831 INFO:tasks.workunit.client.0.smithi028.stdout: "optimal_tunables": 1, 2021-05-12T09:08:03.831 INFO:tasks.workunit.client.0.smithi028.stdout: "legacy_tunables": 0, 2021-05-12T09:08:03.831 INFO:tasks.workunit.client.0.smithi028.stdout: "minimum_required_version": "jewel", 2021-05-12T09:08:03.831 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables": 1, 2021-05-12T09:08:03.832 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables2": 1, 2021-05-12T09:08:03.832 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v2_rules": 0, 2021-05-12T09:08:03.832 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables3": 1, 2021-05-12T09:08:03.832 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v3_rules": 0, 2021-05-12T09:08:03.833 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v4_buckets": 1, 2021-05-12T09:08:03.833 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables5": 1, 2021-05-12T09:08:03.833 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v5_rules": 0 2021-05-12T09:08:03.833 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.834 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_args": {} 2021-05-12T09:08:03.834 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.834 INFO:tasks.workunit.client.0.smithi028.stdout: "fsmap": { 2021-05-12T09:08:03.834 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 1, 2021-05-12T09:08:03.834 INFO:tasks.workunit.client.0.smithi028.stdout: "default_fscid": -1, 2021-05-12T09:08:03.835 INFO:tasks.workunit.client.0.smithi028.stdout: "compat": { 2021-05-12T09:08:03.835 INFO:tasks.workunit.client.0.smithi028.stdout: "compat": {}, 2021-05-12T09:08:03.835 INFO:tasks.workunit.client.0.smithi028.stdout: "ro_compat": {}, 2021-05-12T09:08:03.835 INFO:tasks.workunit.client.0.smithi028.stdout: "incompat": { 2021-05-12T09:08:03.836 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_1": "base v0.20", 2021-05-12T09:08:03.836 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_2": "client writeable ranges", 2021-05-12T09:08:03.836 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T09:08:03.836 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_4": "dir inode in separate object", 2021-05-12T09:08:03.836 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T09:08:03.837 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T09:08:03.837 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_8": "no anchor table", 2021-05-12T09:08:03.837 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_9": "file layout v2", 2021-05-12T09:08:03.837 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_10": "snaprealm v2" 2021-05-12T09:08:03.838 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.838 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.838 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_flags": { 2021-05-12T09:08:03.838 INFO:tasks.workunit.client.0.smithi028.stdout: "enable_multiple": true, 2021-05-12T09:08:03.838 INFO:tasks.workunit.client.0.smithi028.stdout: "ever_enabled_multiple": true 2021-05-12T09:08:03.839 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.839 INFO:tasks.workunit.client.0.smithi028.stdout: "standbys": [], 2021-05-12T09:08:03.839 INFO:tasks.workunit.client.0.smithi028.stdout: "filesystems": [] 2021-05-12T09:08:03.839 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.839 INFO:tasks.workunit.client.0.smithi028.stdout: "mdsmap_first_committed": 1, 2021-05-12T09:08:03.840 INFO:tasks.workunit.client.0.smithi028.stdout: "mdsmap_last_committed": 1, 2021-05-12T09:08:03.840 INFO:tasks.workunit.client.0.smithi028.stdout: "auth": { 2021-05-12T09:08:03.840 INFO:tasks.workunit.client.0.smithi028.stdout: "first_committed": 1, 2021-05-12T09:08:03.840 INFO:tasks.workunit.client.0.smithi028.stdout: "last_committed": 5, 2021-05-12T09:08:03.840 INFO:tasks.workunit.client.0.smithi028.stdout: "num_secrets": 3 2021-05-12T09:08:03.841 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.841 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 2, 2021-05-12T09:08:03.841 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_active": 2, 2021-05-12T09:08:03.841 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_unknown": 0, 2021-05-12T09:08:03.842 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osd": 3, 2021-05-12T09:08:03.842 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_sum": { 2021-05-12T09:08:03.842 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:08:03.842 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 507227, 2021-05-12T09:08:03.843 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 102, 2021-05-12T09:08:03.843 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:08:03.843 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 306, 2021-05-12T09:08:03.843 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:08:03.844 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:08:03.844 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 102, 2021-05-12T09:08:03.844 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:08:03.844 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:08:03.844 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 102, 2021-05-12T09:08:03.845 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:08:03.845 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 76, 2021-05-12T09:08:03.845 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 64, 2021-05-12T09:08:03.845 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 196, 2021-05-12T09:08:03.846 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 1211, 2021-05-12T09:08:03.846 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:08:03.846 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:08:03.846 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:08:03.847 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:08:03.847 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:08:03.847 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:08:03.847 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:08:03.848 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:08:03.848 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:08:03.848 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:08:03.848 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:08:03.848 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:08:03.849 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:08:03.849 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:08:03.849 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:08:03.849 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:08:03.850 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:08:03.850 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:08:03.850 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:08:03.850 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:08:03.851 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:08:03.851 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:08:03.851 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:08:03.851 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:08:03.852 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:08:03.852 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.852 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:08:03.852 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:08:03.852 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:08:03.853 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:08:03.853 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 0, 2021-05-12T09:08:03.853 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 0, 2021-05-12T09:08:03.853 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:08:03.854 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:08:03.854 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:08:03.854 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:08:03.854 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:08:03.854 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.855 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 31, 2021-05-12T09:08:03.855 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 31, 2021-05-12T09:08:03.855 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 4, 2021-05-12T09:08:03.856 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 4, 2021-05-12T09:08:03.856 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 0 2021-05-12T09:08:03.856 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.856 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_sum": { 2021-05-12T09:08:03.856 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 0, 2021-05-12T09:08:03.857 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 0, 2021-05-12T09:08:03.857 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pgs": 6, 2021-05-12T09:08:03.857 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osds": 3, 2021-05-12T09:08:03.857 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_osds": 3, 2021-05-12T09:08:03.858 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T09:08:03.858 INFO:tasks.workunit.client.0.smithi028.stdout: "kb": 314572800, 2021-05-12T09:08:03.858 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used": 21460, 2021-05-12T09:08:03.858 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_data": 3572, 2021-05-12T09:08:03.858 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_omap": 2, 2021-05-12T09:08:03.859 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_meta": 17853, 2021-05-12T09:08:03.859 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_avail": 314551340, 2021-05-12T09:08:03.859 INFO:tasks.workunit.client.0.smithi028.stdout: "statfs": { 2021-05-12T09:08:03.860 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 322122547200, 2021-05-12T09:08:03.860 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 322100572160, 2021-05-12T09:08:03.860 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:08:03.860 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 3657728, 2021-05-12T09:08:03.860 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1864151, 2021-05-12T09:08:03.861 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:08:03.861 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:08:03.861 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:08:03.861 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 2160, 2021-05-12T09:08:03.863 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 18282384 2021-05-12T09:08:03.863 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.864 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_peers": [], 2021-05-12T09:08:03.864 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_trim_queue_len": 0, 2021-05-12T09:08:03.864 INFO:tasks.workunit.client.0.smithi028.stdout: "num_snap_trimming": 0, 2021-05-12T09:08:03.864 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shards_repaired": 0, 2021-05-12T09:08:03.865 INFO:tasks.workunit.client.0.smithi028.stdout: "op_queue_age_hist": { 2021-05-12T09:08:03.866 INFO:tasks.workunit.client.0.smithi028.stdout: "histogram": [], 2021-05-12T09:08:03.866 INFO:tasks.workunit.client.0.smithi028.stdout: "upper_bound": 1 2021-05-12T09:08:03.867 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.867 INFO:tasks.workunit.client.0.smithi028.stdout: "perf_stat": { 2021-05-12T09:08:03.867 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ms": 0, 2021-05-12T09:08:03.867 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ms": 0, 2021-05-12T09:08:03.868 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ns": 0, 2021-05-12T09:08:03.868 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ns": 0 2021-05-12T09:08:03.868 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.868 INFO:tasks.workunit.client.0.smithi028.stdout: "alerts": [], 2021-05-12T09:08:03.869 INFO:tasks.workunit.client.0.smithi028.stdout: "network_ping_times": [] 2021-05-12T09:08:03.869 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.869 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_sum_by_class": { 2021-05-12T09:08:03.869 INFO:tasks.workunit.client.0.smithi028.stdout: "hdd": { 2021-05-12T09:08:03.869 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 0, 2021-05-12T09:08:03.870 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 0, 2021-05-12T09:08:03.870 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pgs": 6, 2021-05-12T09:08:03.870 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osds": 3, 2021-05-12T09:08:03.870 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_osds": 3, 2021-05-12T09:08:03.870 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T09:08:03.871 INFO:tasks.workunit.client.0.smithi028.stdout: "kb": 314572800, 2021-05-12T09:08:03.871 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used": 21460, 2021-05-12T09:08:03.871 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_data": 3572, 2021-05-12T09:08:03.871 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_omap": 2, 2021-05-12T09:08:03.872 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_meta": 17853, 2021-05-12T09:08:03.872 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_avail": 314551340, 2021-05-12T09:08:03.872 INFO:tasks.workunit.client.0.smithi028.stdout: "statfs": { 2021-05-12T09:08:03.872 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 322122547200, 2021-05-12T09:08:03.872 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 322100572160, 2021-05-12T09:08:03.873 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:08:03.873 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 3657728, 2021-05-12T09:08:03.873 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1864151, 2021-05-12T09:08:03.873 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:08:03.873 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:08:03.874 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:08:03.874 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 2160, 2021-05-12T09:08:03.874 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 18282384 2021-05-12T09:08:03.874 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.874 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_peers": [], 2021-05-12T09:08:03.875 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_trim_queue_len": 0, 2021-05-12T09:08:03.875 INFO:tasks.workunit.client.0.smithi028.stdout: "num_snap_trimming": 0, 2021-05-12T09:08:03.876 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shards_repaired": 0, 2021-05-12T09:08:03.876 INFO:tasks.workunit.client.0.smithi028.stdout: "op_queue_age_hist": { 2021-05-12T09:08:03.876 INFO:tasks.workunit.client.0.smithi028.stdout: "histogram": [], 2021-05-12T09:08:03.876 INFO:tasks.workunit.client.0.smithi028.stdout: "upper_bound": 1 2021-05-12T09:08:03.877 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.877 INFO:tasks.workunit.client.0.smithi028.stdout: "perf_stat": { 2021-05-12T09:08:03.877 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ms": 0, 2021-05-12T09:08:03.877 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ms": 0, 2021-05-12T09:08:03.878 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ns": 0, 2021-05-12T09:08:03.878 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ns": 0 2021-05-12T09:08:03.878 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.878 INFO:tasks.workunit.client.0.smithi028.stdout: "alerts": [], 2021-05-12T09:08:03.879 INFO:tasks.workunit.client.0.smithi028.stdout: "network_ping_times": [] 2021-05-12T09:08:03.879 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.879 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.879 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_stats": [ 2021-05-12T09:08:03.879 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.880 INFO:tasks.workunit.client.0.smithi028.stdout: "poolid": 2, 2021-05-12T09:08:03.880 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 1, 2021-05-12T09:08:03.880 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:08:03.880 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 47947, 2021-05-12T09:08:03.881 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 100, 2021-05-12T09:08:03.881 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:08:03.881 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 300, 2021-05-12T09:08:03.881 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:08:03.882 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:08:03.882 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 100, 2021-05-12T09:08:03.882 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:08:03.882 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:08:03.883 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 100, 2021-05-12T09:08:03.883 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:08:03.883 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 0, 2021-05-12T09:08:03.883 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 0, 2021-05-12T09:08:03.884 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 101, 2021-05-12T09:08:03.884 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 101, 2021-05-12T09:08:03.884 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:08:03.884 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:08:03.885 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:08:03.885 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:08:03.885 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:08:03.885 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:08:03.886 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:08:03.886 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:08:03.886 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:08:03.886 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:08:03.887 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:08:03.887 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:08:03.887 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:08:03.887 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:08:03.887 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:08:03.888 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:08:03.888 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:08:03.888 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:08:03.888 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:08:03.888 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:08:03.889 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:08:03.889 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:08:03.889 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:08:03.889 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:08:03.890 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:08:03.890 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.890 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:08:03.890 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:08:03.890 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:08:03.891 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:08:03.891 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 1220608, 2021-05-12T09:08:03.891 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 142522, 2021-05-12T09:08:03.891 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:08:03.892 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:08:03.892 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:08:03.892 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:08:03.892 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:08:03.893 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.893 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 17, 2021-05-12T09:08:03.893 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 17, 2021-05-12T09:08:03.893 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 2, 2021-05-12T09:08:03.893 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 2, 2021-05-12T09:08:03.894 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 3 2021-05-12T09:08:03.894 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.894 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.895 INFO:tasks.workunit.client.0.smithi028.stdout: "poolid": 1, 2021-05-12T09:08:03.895 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 1, 2021-05-12T09:08:03.895 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:08:03.895 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 459280, 2021-05-12T09:08:03.896 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 2, 2021-05-12T09:08:03.896 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:08:03.896 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 6, 2021-05-12T09:08:03.896 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:08:03.896 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:08:03.897 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 2, 2021-05-12T09:08:03.897 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:08:03.897 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:08:03.897 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 2, 2021-05-12T09:08:03.897 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:08:03.898 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 76, 2021-05-12T09:08:03.898 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 64, 2021-05-12T09:08:03.898 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 95, 2021-05-12T09:08:03.898 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 1110, 2021-05-12T09:08:03.899 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:08:03.899 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:08:03.899 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:08:03.899 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:08:03.899 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:08:03.900 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:08:03.900 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:08:03.900 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:08:03.900 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:08:03.901 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:08:03.901 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:08:03.901 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:08:03.901 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:08:03.902 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:08:03.902 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:08:03.902 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:08:03.902 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:08:03.902 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:08:03.903 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:08:03.903 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:08:03.903 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:08:03.903 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:08:03.904 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:08:03.904 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:08:03.904 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:08:03.904 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.904 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:08:03.905 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:08:03.905 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:08:03.905 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:08:03.905 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 1388544, 2021-05-12T09:08:03.906 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1377840, 2021-05-12T09:08:03.906 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:08:03.906 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:08:03.906 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:08:03.907 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:08:03.907 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:08:03.907 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.907 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 14, 2021-05-12T09:08:03.907 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 14, 2021-05-12T09:08:03.908 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 2, 2021-05-12T09:08:03.908 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 2, 2021-05-12T09:08:03.908 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 3 2021-05-12T09:08:03.909 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.909 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.909 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_stats": [ 2021-05-12T09:08:03.909 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.909 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:08:03.910 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 167503724563 2021-05-12T09:08:03.910 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.910 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.910 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:08:03.911 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 176093659155 2021-05-12T09:08:03.911 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.911 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.911 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:08:03.912 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 64424509455 2021-05-12T09:08:03.912 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.912 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.912 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_by_state": [ 2021-05-12T09:08:03.913 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.913 INFO:tasks.workunit.client.0.smithi028.stdout: "state": "active+undersized+degraded", 2021-05-12T09:08:03.913 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 2 2021-05-12T09:08:03.913 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.914 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.914 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_by_osd": [ 2021-05-12T09:08:03.914 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.914 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:08:03.915 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:08:03.915 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 0, 2021-05-12T09:08:03.915 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:08:03.916 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.916 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.916 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:08:03.916 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 2, 2021-05-12T09:08:03.917 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 2, 2021-05-12T09:08:03.917 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:08:03.917 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.917 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.918 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:08:03.918 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:08:03.918 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 2, 2021-05-12T09:08:03.918 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:08:03.918 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.919 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.919 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": [ 2021-05-12T09:08:03.919 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.919 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 1, 2021-05-12T09:08:03.920 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": {} 2021-05-12T09:08:03.920 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.920 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:08:03.920 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 2, 2021-05-12T09:08:03.921 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": {} 2021-05-12T09:08:03.921 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.921 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:08:03.921 INFO:tasks.workunit.client.0.smithi028.stdout: "servicemap": { 2021-05-12T09:08:03.922 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 7, 2021-05-12T09:08:03.922 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:07:51.830128+0000", 2021-05-12T09:08:03.922 INFO:tasks.workunit.client.0.smithi028.stdout: "services": {} 2021-05-12T09:08:03.922 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:08:03.922 INFO:tasks.workunit.client.0.smithi028.stdout: "paxos": { 2021-05-12T09:08:03.923 INFO:tasks.workunit.client.0.smithi028.stdout: "first_committed": 1, 2021-05-12T09:08:03.923 INFO:tasks.workunit.client.0.smithi028.stdout: "last_committed": 459, 2021-05-12T09:08:03.923 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pn": 0, 2021-05-12T09:08:03.923 INFO:tasks.workunit.client.0.smithi028.stdout: "accepted_pn": 0 2021-05-12T09:08:03.924 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:08:03.924 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:08:03.925 INFO:tasks.workunit.client.0.smithi028.stderr:report 422304533 2021-05-12T09:08:03.925 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T09:08:03.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2021-05-12T09:08:03.926 INFO:tasks.workunit.client.0.smithi028.stdout:delay recovery 2021-05-12T09:08:03.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2021-05-12T09:08:03.988 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:08:03.989 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T09:06:53.004909+0000 47'101 47:122 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:05:59.322736+0000 0'0 2021-05-12T09:05:59.322736+0000 0 2021-05-12T09:08:03.989 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:06:52.929459+0000 32'62 47:115 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:05:55.908046+0000 0'0 2021-05-12T09:05:55.908046+0000 0 2021-05-12T09:08:03.989 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:08:03.990 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:08:03.991 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:08:04.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2021-05-12T09:08:04.001 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2021-05-12T09:08:04.001 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T09:08:04.002 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T09:08:04.002 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:08:04.002 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:08:04.002 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:08:04.003 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.0.asok 2021-05-12T09:08:04.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2021-05-12T09:08:04.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.0.asok set_recovery_delay 100000 2021-05-12T09:08:04.101 INFO:tasks.workunit.client.0.smithi028.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T09:08:04.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2021-05-12T09:08:04.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.1 2021-05-12T09:08:04.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T09:08:04.112 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T09:08:04.112 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:08:04.112 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:08:04.112 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:08:04.112 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.1.asok 2021-05-12T09:08:04.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2021-05-12T09:08:04.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.1.asok set_recovery_delay 100000 2021-05-12T09:08:04.212 INFO:tasks.workunit.client.0.smithi028.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T09:08:04.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 2' 2021-05-12T09:08:04.221 INFO:tasks.workunit.client.0.smithi028.stdout:revive divergent 2 2021-05-12T09:08:04.221 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2021-05-12T09:08:04.585 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:08:04.585 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T09:06:53.004909+0000 47'101 47:122 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:05:59.322736+0000 0'0 2021-05-12T09:05:59.322736+0000 0 2021-05-12T09:08:04.585 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:06:52.929459+0000 32'62 47:115 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:05:55.908046+0000 0'0 2021-05-12T09:05:55.908046+0000 0 2021-05-12T09:08:04.586 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:08:04.586 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:08:04.587 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:08:04.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2021-05-12T09:08:05.270 INFO:tasks.workunit.client.0.smithi028.stderr:noup is set 2021-05-12T09:08:05.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 2 2021-05-12T09:08:05.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:08:05.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:08:05.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T09:08:05.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:08:05.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T09:08:05.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:08:05.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:08:05.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:08:05.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:08:05.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T09:08:05.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T09:08:05.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:08:05.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:08:05.284 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:08:05.284 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:08:05.284 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:08:05.284 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:08:05.285 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:08:05.285 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:08:05.285 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:08:05.286 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:08:05.287 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:08:05.287 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:08:05.287 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:08:05.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:08:05.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:08:05.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:08:05.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:08:05.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:08:05.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:08:05.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T09:08:05.290 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:08:05.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T09:08:05.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:08:05.291 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T09:08:05.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T09:08:05.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:08:05.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:08:05.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:08:05.310 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:05.308+0000 7f887c5500c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:08:05.324 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:05.323+0000 7f887c5500c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:08:05.325 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:05.324+0000 7f887c5500c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:08:05.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: return 0 2021-05-12T09:08:05.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2021-05-12T09:08:05.841 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:05.840+0000 7f887c5500c0 -1 Falling back to public interface 2021-05-12T09:08:08.539 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:08.538+0000 7f887c5500c0 -1 osd.2 32 log_to_monitors {default=true} 2021-05-12T09:08:10.671 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2021-05-12T09:08:10.671 INFO:tasks.workunit.client.0.smithi028.stdout:delay recovery divergent 2021-05-12T09:08:10.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2021-05-12T09:08:11.050 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:08:11.050 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T09:06:53.004909+0000 47'101 47:123 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:05:59.322736+0000 0'0 2021-05-12T09:05:59.322736+0000 0 2021-05-12T09:08:11.050 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:06:52.929459+0000 32'62 47:116 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:05:55.908046+0000 0'0 2021-05-12T09:05:55.908046+0000 0 2021-05-12T09:08:11.051 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:08:11.051 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:08:11.052 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:08:11.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.2 2021-05-12T09:08:11.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T09:08:11.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T09:08:11.062 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:08:11.063 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:08:11.063 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:08:11.063 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.2.asok 2021-05-12T09:08:11.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2021-05-12T09:08:11.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.2.asok set_recovery_delay 100000 2021-05-12T09:08:11.162 INFO:tasks.workunit.client.0.smithi028.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T09:08:11.170 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2021-05-12T09:08:12.021 INFO:tasks.workunit.client.0.smithi028.stderr:noup is unset 2021-05-12T09:08:12.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2021-05-12T09:08:12.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:08:12.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:08:12.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:08:12.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:08:12.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:08:12.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:08:12.037 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:08:12.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:08:12.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:08:12.479 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 39 up_thru 45 down_at 33 last_clean_interval [5,32) [v2:127.0.0.1:6802/49441,v1:127.0.0.1:6803/49441] [v2:127.0.0.1:6804/49441,v1:127.0.0.1:6805/49441] exists,up c5a51d7a-897f-41a2-97c3-e9ef91d860e1 2021-05-12T09:08:12.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:08:12.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:08:12.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:08:12.482 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2021-05-12T09:08:12.482 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:08:12.482 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:08:12.482 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:08:12.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:08:12.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:08:12.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:08:12.484 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:08:12.484 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:08:12.484 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:08:12.865 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 41 up_thru 52 down_at 34 last_clean_interval [10,32) [v2:127.0.0.1:6810/49998,v1:127.0.0.1:6811/49998] [v2:127.0.0.1:6812/49998,v1:127.0.0.1:6813/49998] exists,up e003828b-f9e6-48de-bd5a-128e4af3e5dc 2021-05-12T09:08:12.866 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:08:12.866 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:08:12.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:08:12.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2021-05-12T09:08:12.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:08:12.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:08:12.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:08:12.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:08:12.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:08:12.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:08:12.869 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:08:12.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:08:12.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:08:13.258 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 52 up_thru 52 down_at 35 last_clean_interval [15,32) [v2:127.0.0.1:6818/52672,v1:127.0.0.1:6819/52672] [v2:127.0.0.1:6820/52672,v1:127.0.0.1:6821/52672] exists,up 10e9d857-0855-44ba-8b8e-8d823ab9204a 2021-05-12T09:08:13.259 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:08:13.259 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:08:13.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:08:13.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2021-05-12T09:08:13.635 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:08:13.635 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-05-12T09:06:53.004909+0000 47'101 49:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:05:59.322736+0000 0'0 2021-05-12T09:05:59.322736+0000 0 2021-05-12T09:08:13.635 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:06:52.929459+0000 32'62 49:117 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:05:55.908046+0000 0'0 2021-05-12T09:05:55.908046+0000 0 2021-05-12T09:08:13.636 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:08:13.636 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:08:13.637 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:08:13.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2021-05-12T09:08:13.647 INFO:tasks.workunit.client.0.smithi028.stdout:wait for peering 2021-05-12T09:08:13.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2021-05-12T09:08:14.008 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:08:14.009 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2021-05-12T09:08:12.623789+0000 47'101 54:129 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:05:59.322736+0000 0'0 2021-05-12T09:05:59.322736+0000 0 2021-05-12T09:08:14.009 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:06:52.929459+0000 32'62 49:117 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:05:55.908046+0000 0'0 2021-05-12T09:05:55.908046+0000 0 2021-05-12T09:08:14.009 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:08:14.009 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:08:14.010 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:08:14.019 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2021-05-12T09:08:21.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 2' 2021-05-12T09:08:21.085 INFO:tasks.workunit.client.0.smithi028.stdout:killing divergent 2 2021-05-12T09:08:21.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2021-05-12T09:08:21.461 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:08:21.461 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2021-05-12T09:08:12.623789+0000 47'101 54:129 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:05:59.322736+0000 0'0 2021-05-12T09:05:59.322736+0000 0 2021-05-12T09:08:21.461 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:08:12.537912+0000 32'62 54:129 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:05:55.908046+0000 0'0 2021-05-12T09:05:55.908046+0000 0 2021-05-12T09:08:21.461 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:08:21.462 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:08:21.463 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:08:21.472 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.2 2021-05-12T09:08:21.473 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:08:21.473 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:08:21.473 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:08:21.474 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:08:21.474 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:08:21.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:08:21.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.29719.out 2021-05-12T09:08:21.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 2 --op export-remove --pgid 2.0 --file td/divergent-priors/exp.29719.out 2021-05-12T09:08:21.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T09:08:21.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T09:08:21.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=2 2021-05-12T09:08:21.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T09:08:21.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2021-05-12T09:08:21.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op export-remove --pgid 2.0 --file td/divergent-priors/exp.29719.out 2021-05-12T09:08:22.369 INFO:tasks.workunit.client.0.smithi028.stderr:Exporting 2.0 info 2.0( v 54'102 lc 0'0 (32'90,54'102] local-lis/les=52/54 n=101 ec=26/26 lis/c=52/26 les/c/f=54/27/0 sis=52 pruub=8.384708405s) 2021-05-12T09:08:22.377 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:023dd3d2:::existing_12:head# 2021-05-12T09:08:22.404 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:04c40a0c:::existing_93:head# 2021-05-12T09:08:22.425 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:064ff5fb:::existing_36:head# 2021-05-12T09:08:22.431 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:091f9f1d:::existing_94:head# 2021-05-12T09:08:22.433 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:0a865a93:::existing_54:head# 2021-05-12T09:08:22.434 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:0fed513c:::existing_41:head# 2021-05-12T09:08:22.434 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:13b9e0b8:::existing_92:head# 2021-05-12T09:08:22.435 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:169157fa:::existing_81:head# 2021-05-12T09:08:22.435 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:17731382:::existing_58:head# 2021-05-12T09:08:22.436 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:18a45a39:::existing_43:head# 2021-05-12T09:08:22.436 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:1c160f01:::existing_66:head# 2021-05-12T09:08:22.437 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:1dd6c249:::existing_97:head# 2021-05-12T09:08:22.437 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:227e0c77:::existing_7:head# 2021-05-12T09:08:22.438 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:23e1b063:::existing_59:head# 2021-05-12T09:08:22.439 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:28566edb:::existing_78:head# 2021-05-12T09:08:22.440 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:2a101182:::existing_20:head# 2021-05-12T09:08:22.440 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:2d1519da:::existing_96:head# 2021-05-12T09:08:22.441 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:3078b87a:::existing_86:head# 2021-05-12T09:08:22.441 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:30bde01b:::existing_15:head# 2021-05-12T09:08:22.441 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:317a7e19:::existing_56:head# 2021-05-12T09:08:22.442 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:38b9c7fa:::existing_33:head# 2021-05-12T09:08:22.442 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:3e16d84b:::existing_48:head# 2021-05-12T09:08:22.443 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:45fe8203:::existing_75:head# 2021-05-12T09:08:22.443 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:4e82c9ba:::existing_50:head# 2021-05-12T09:08:22.444 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:5298dcdb:::existing_57:head# 2021-05-12T09:08:22.444 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:52ebaa44:::existing_32:head# 2021-05-12T09:08:22.445 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:53588912:::existing_9:head# 2021-05-12T09:08:22.445 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:5b7e2de6:::existing_84:head# 2021-05-12T09:08:22.447 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:5da0255c:::existing_13:head# 2021-05-12T09:08:22.447 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:5da2dab8:::existing_53:head# 2021-05-12T09:08:22.447 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:5ee3514b:::existing_30:head# 2021-05-12T09:08:22.447 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:602f83fe:::foo:head# 2021-05-12T09:08:22.448 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:60426cc7:::existing_62:head# 2021-05-12T09:08:22.448 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:628918ca:::existing_79:head# 2021-05-12T09:08:22.449 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:63f8ab2d:::existing_77:head# 2021-05-12T09:08:22.449 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:67e1780f:::existing_27:head# 2021-05-12T09:08:22.450 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:68143ca1:::existing_71:head# 2021-05-12T09:08:22.450 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:6bac9509:::existing_29:head# 2021-05-12T09:08:22.450 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:6bbec798:::existing_45:head# 2021-05-12T09:08:22.451 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:6cb23a40:::existing_65:head# 2021-05-12T09:08:22.451 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:7055ec5e:::existing_83:head# 2021-05-12T09:08:22.452 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:70a54e2b:::existing_87:head# 2021-05-12T09:08:22.453 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:750ace21:::existing_19:head# 2021-05-12T09:08:22.453 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:78623b1d:::existing_6:head# 2021-05-12T09:08:22.453 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:79881152:::existing_63:head# 2021-05-12T09:08:22.454 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:7a80c0d5:::existing_99:head# 2021-05-12T09:08:22.454 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:81490677:::existing_76:head# 2021-05-12T09:08:22.455 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:85ec8e5d:::existing_85:head# 2021-05-12T09:08:22.455 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:8e2abbf0:::existing_49:head# 2021-05-12T09:08:22.456 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:8ee0fe0f:::existing_52:head# 2021-05-12T09:08:22.456 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:923c5d4d:::existing_5:head# 2021-05-12T09:08:22.457 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:92731e71:::existing_89:head# 2021-05-12T09:08:22.457 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:928205cf:::existing_82:head# 2021-05-12T09:08:22.458 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:93be2267:::existing_24:head# 2021-05-12T09:08:22.458 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:99ceda2c:::existing_44:head# 2021-05-12T09:08:22.459 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:9dd917ee:::existing_26:head# 2021-05-12T09:08:22.459 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:a28805a6:::existing_72:head# 2021-05-12T09:08:22.459 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:a294a8f6:::existing_16:head# 2021-05-12T09:08:22.460 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:a2f5473a:::existing_42:head# 2021-05-12T09:08:22.460 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:a504a533:::existing_90:head# 2021-05-12T09:08:22.461 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:a691337e:::existing_61:head# 2021-05-12T09:08:22.462 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:a6e8d7ac:::existing_28:head# 2021-05-12T09:08:22.462 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:a9fde86e:::existing_18:head# 2021-05-12T09:08:22.463 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:aa309faa:::existing_68:head# 2021-05-12T09:08:22.463 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:ae40b0ed:::existing_31:head# 2021-05-12T09:08:22.464 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:b279b1fe:::existing_10:head# 2021-05-12T09:08:22.464 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:b343e277:::existing_88:head# 2021-05-12T09:08:22.465 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:b7130e90:::existing_80:head# 2021-05-12T09:08:22.465 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:bc6f7309:::existing_70:head# 2021-05-12T09:08:22.466 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:be8489b7:::existing_38:head# 2021-05-12T09:08:22.466 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:beb4e5e3:::existing_47:head# 2021-05-12T09:08:22.467 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:bebec877:::existing_74:head# 2021-05-12T09:08:22.467 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:c04347d8:::existing_60:head# 2021-05-12T09:08:22.467 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:c156033e:::existing_100:head# 2021-05-12T09:08:22.468 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:c19f5a8e:::existing_8:head# 2021-05-12T09:08:22.468 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:c3e6ebe4:::existing_17:head# 2021-05-12T09:08:22.469 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:c6c5a7c2:::existing_39:head# 2021-05-12T09:08:22.469 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:cc1d0abf:::existing_95:head# 2021-05-12T09:08:22.470 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:ccb3865b:::existing_73:head# 2021-05-12T09:08:22.470 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:cd84e19e:::existing_40:head# 2021-05-12T09:08:22.471 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:d2cd64ce:::existing_37:head# 2021-05-12T09:08:22.471 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:d41a25c9:::existing_22:head# 2021-05-12T09:08:22.472 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:d7f088da:::existing_91:head# 2021-05-12T09:08:22.473 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:d9cbd1d6:::existing_64:head# 2021-05-12T09:08:22.473 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:dcc4ff82:::existing_98:head# 2021-05-12T09:08:22.474 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:dd677bd3:::existing_35:head# 2021-05-12T09:08:22.474 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:dd6d90e7:::existing_46:head# 2021-05-12T09:08:22.474 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:e0e22077:::existing_21:head# 2021-05-12T09:08:22.479 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:e255b710:::existing_55:head# 2021-05-12T09:08:22.479 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:eff1d5db:::existing_14:head# 2021-05-12T09:08:22.480 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:f1b156bf:::existing_23:head# 2021-05-12T09:08:22.480 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:f5ce4199:::existing_34:head# 2021-05-12T09:08:22.480 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:f6809456:::existing_11:head# 2021-05-12T09:08:22.480 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:f699a2a9:::existing_67:head# 2021-05-12T09:08:22.481 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:f9baac43:::existing_25:head# 2021-05-12T09:08:22.481 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:fb76688e:::existing_51:head# 2021-05-12T09:08:22.481 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:fbb92eec:::existing_69:head# 2021-05-12T09:08:22.482 INFO:tasks.workunit.client.0.smithi028.stderr:Export successful 2021-05-12T09:08:22.483 INFO:tasks.workunit.client.0.smithi028.stdout: marking collection for removal 2021-05-12T09:08:22.483 INFO:tasks.workunit.client.0.smithi028.stdout:setting '_remove' omap key 2021-05-12T09:08:22.483 INFO:tasks.workunit.client.0.smithi028.stdout:finish_remove_pgs 2.0_head removing 2.0 2021-05-12T09:08:22.702 INFO:tasks.workunit.client.0.smithi028.stderr:Remove successful 2021-05-12T09:08:22.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 2 --op import --file td/divergent-priors/exp.29719.out 2021-05-12T09:08:23.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T09:08:23.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T09:08:23.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=2 2021-05-12T09:08:23.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T09:08:23.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2021-05-12T09:08:23.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op import --file td/divergent-priors/exp.29719.out 2021-05-12T09:08:23.813 INFO:tasks.workunit.client.0.smithi028.stdout:get_pg_num_history pg_num_history pg_num_history(e54 pg_nums {1={16=1},2={26=1}} deleted_pools ) 2021-05-12T09:08:23.813 INFO:tasks.workunit.client.0.smithi028.stdout:Importing pgid 2.0 2021-05-12T09:08:23.814 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:023dd3d2:::existing_12:head# 2021-05-12T09:08:23.814 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:23.880 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:04c40a0c:::existing_93:head# 2021-05-12T09:08:23.889 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:23.946 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:064ff5fb:::existing_36:head# 2021-05-12T09:08:23.947 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:23.996 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:091f9f1d:::existing_94:head# 2021-05-12T09:08:23.997 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.030 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:0a865a93:::existing_54:head# 2021-05-12T09:08:24.030 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.064 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:0fed513c:::existing_41:head# 2021-05-12T09:08:24.064 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.097 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:13b9e0b8:::existing_92:head# 2021-05-12T09:08:24.097 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.131 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:169157fa:::existing_81:head# 2021-05-12T09:08:24.131 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.164 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:17731382:::existing_58:head# 2021-05-12T09:08:24.164 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.206 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:18a45a39:::existing_43:head# 2021-05-12T09:08:24.206 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.289 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:1c160f01:::existing_66:head# 2021-05-12T09:08:24.290 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.373 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:1dd6c249:::existing_97:head# 2021-05-12T09:08:24.373 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.415 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:227e0c77:::existing_7:head# 2021-05-12T09:08:24.415 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.448 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:23e1b063:::existing_59:head# 2021-05-12T09:08:24.449 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.482 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:28566edb:::existing_78:head# 2021-05-12T09:08:24.482 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.515 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:2a101182:::existing_20:head# 2021-05-12T09:08:24.516 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.541 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:2d1519da:::existing_96:head# 2021-05-12T09:08:24.541 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.566 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:3078b87a:::existing_86:head# 2021-05-12T09:08:24.566 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.591 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:30bde01b:::existing_15:head# 2021-05-12T09:08:24.591 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.616 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:317a7e19:::existing_56:head# 2021-05-12T09:08:24.616 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.641 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:38b9c7fa:::existing_33:head# 2021-05-12T09:08:24.641 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.666 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:3e16d84b:::existing_48:head# 2021-05-12T09:08:24.666 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.691 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:45fe8203:::existing_75:head# 2021-05-12T09:08:24.691 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.716 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:4e82c9ba:::existing_50:head# 2021-05-12T09:08:24.717 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.741 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:5298dcdb:::existing_57:head# 2021-05-12T09:08:24.742 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.767 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:52ebaa44:::existing_32:head# 2021-05-12T09:08:24.767 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.792 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:53588912:::existing_9:head# 2021-05-12T09:08:24.792 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.817 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:5b7e2de6:::existing_84:head# 2021-05-12T09:08:24.817 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.842 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:5da0255c:::existing_13:head# 2021-05-12T09:08:24.842 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.867 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:5da2dab8:::existing_53:head# 2021-05-12T09:08:24.867 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.892 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:5ee3514b:::existing_30:head# 2021-05-12T09:08:24.892 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.917 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:602f83fe:::foo:head# 2021-05-12T09:08:24.918 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.959 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:60426cc7:::existing_62:head# 2021-05-12T09:08:24.960 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:24.984 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:628918ca:::existing_79:head# 2021-05-12T09:08:24.985 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.009 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:63f8ab2d:::existing_77:head# 2021-05-12T09:08:25.010 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.035 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:67e1780f:::existing_27:head# 2021-05-12T09:08:25.035 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.060 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:68143ca1:::existing_71:head# 2021-05-12T09:08:25.060 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.085 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:6bac9509:::existing_29:head# 2021-05-12T09:08:25.085 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.110 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:6bbec798:::existing_45:head# 2021-05-12T09:08:25.110 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.169 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:6cb23a40:::existing_65:head# 2021-05-12T09:08:25.169 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.202 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:7055ec5e:::existing_83:head# 2021-05-12T09:08:25.202 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.236 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:70a54e2b:::existing_87:head# 2021-05-12T09:08:25.236 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.269 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:750ace21:::existing_19:head# 2021-05-12T09:08:25.269 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.302 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:78623b1d:::existing_6:head# 2021-05-12T09:08:25.302 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.336 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:79881152:::existing_63:head# 2021-05-12T09:08:25.336 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.369 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:7a80c0d5:::existing_99:head# 2021-05-12T09:08:25.370 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.403 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:81490677:::existing_76:head# 2021-05-12T09:08:25.403 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.436 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:85ec8e5d:::existing_85:head# 2021-05-12T09:08:25.436 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.469 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:8e2abbf0:::existing_49:head# 2021-05-12T09:08:25.470 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.503 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:8ee0fe0f:::existing_52:head# 2021-05-12T09:08:25.503 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.536 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:923c5d4d:::existing_5:head# 2021-05-12T09:08:25.537 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.570 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:92731e71:::existing_89:head# 2021-05-12T09:08:25.570 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.603 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:928205cf:::existing_82:head# 2021-05-12T09:08:25.604 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.637 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:93be2267:::existing_24:head# 2021-05-12T09:08:25.637 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.670 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:99ceda2c:::existing_44:head# 2021-05-12T09:08:25.670 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.703 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:9dd917ee:::existing_26:head# 2021-05-12T09:08:25.704 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.737 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:a28805a6:::existing_72:head# 2021-05-12T09:08:25.737 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.770 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:a294a8f6:::existing_16:head# 2021-05-12T09:08:25.771 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.804 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:a2f5473a:::existing_42:head# 2021-05-12T09:08:25.804 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.837 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:a504a533:::existing_90:head# 2021-05-12T09:08:25.838 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.871 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:a691337e:::existing_61:head# 2021-05-12T09:08:25.871 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:25.913 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:a6e8d7ac:::existing_28:head# 2021-05-12T09:08:25.913 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.013 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:a9fde86e:::existing_18:head# 2021-05-12T09:08:26.014 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.047 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:aa309faa:::existing_68:head# 2021-05-12T09:08:26.047 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.080 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:ae40b0ed:::existing_31:head# 2021-05-12T09:08:26.080 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.114 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:b279b1fe:::existing_10:head# 2021-05-12T09:08:26.114 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.147 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:b343e277:::existing_88:head# 2021-05-12T09:08:26.147 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.181 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:b7130e90:::existing_80:head# 2021-05-12T09:08:26.181 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.239 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:bc6f7309:::existing_70:head# 2021-05-12T09:08:26.239 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.272 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:be8489b7:::existing_38:head# 2021-05-12T09:08:26.273 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.306 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:beb4e5e3:::existing_47:head# 2021-05-12T09:08:26.306 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.339 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:bebec877:::existing_74:head# 2021-05-12T09:08:26.340 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.373 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:c04347d8:::existing_60:head# 2021-05-12T09:08:26.373 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.415 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:c156033e:::existing_100:head# 2021-05-12T09:08:26.415 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.448 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:c19f5a8e:::existing_8:head# 2021-05-12T09:08:26.448 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.482 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:c3e6ebe4:::existing_17:head# 2021-05-12T09:08:26.482 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.515 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:c6c5a7c2:::existing_39:head# 2021-05-12T09:08:26.515 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.549 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:cc1d0abf:::existing_95:head# 2021-05-12T09:08:26.549 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.590 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:ccb3865b:::existing_73:head# 2021-05-12T09:08:26.591 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.633 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:cd84e19e:::existing_40:head# 2021-05-12T09:08:26.633 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.666 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:d2cd64ce:::existing_37:head# 2021-05-12T09:08:26.666 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.699 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:d41a25c9:::existing_22:head# 2021-05-12T09:08:26.700 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.733 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:d7f088da:::existing_91:head# 2021-05-12T09:08:26.733 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.766 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:d9cbd1d6:::existing_64:head# 2021-05-12T09:08:26.767 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.818 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:dcc4ff82:::existing_98:head# 2021-05-12T09:08:26.819 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.841 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:dd677bd3:::existing_35:head# 2021-05-12T09:08:26.842 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.866 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:dd6d90e7:::existing_46:head# 2021-05-12T09:08:26.867 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.892 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:e0e22077:::existing_21:head# 2021-05-12T09:08:26.892 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.917 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:e255b710:::existing_55:head# 2021-05-12T09:08:26.917 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.942 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:eff1d5db:::existing_14:head# 2021-05-12T09:08:26.942 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.967 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:f1b156bf:::existing_23:head# 2021-05-12T09:08:26.967 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:26.992 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:f5ce4199:::existing_34:head# 2021-05-12T09:08:26.992 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:27.017 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:f6809456:::existing_11:head# 2021-05-12T09:08:27.017 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:27.042 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:f699a2a9:::existing_67:head# 2021-05-12T09:08:27.043 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:27.068 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:f9baac43:::existing_25:head# 2021-05-12T09:08:27.068 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:27.093 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:fb76688e:::existing_51:head# 2021-05-12T09:08:27.093 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:27.118 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:fbb92eec:::existing_69:head# 2021-05-12T09:08:27.118 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:08:27.143 INFO:tasks.workunit.client.0.smithi028.stdout:write_pg epoch 54 info 2.0( v 54'102 lc 0'0 (32'90,54'102] local-lis/les=52/54 n=101 ec=26/26 lis/c=52/26 les/c/f=54/27/0 sis=52 pruub=8.384708405s) 2021-05-12T09:08:27.168 INFO:tasks.workunit.client.0.smithi028.stdout:Import successful 2021-05-12T09:08:27.686 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 2' 2021-05-12T09:08:27.686 INFO:tasks.workunit.client.0.smithi028.stdout:reviving divergent 2 2021-05-12T09:08:27.687 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2021-05-12T09:08:28.063 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:08:28.064 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2021-05-12T09:08:12.623789+0000 47'101 54:129 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:05:59.322736+0000 0'0 2021-05-12T09:05:59.322736+0000 0 2021-05-12T09:08:28.064 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:08:12.537912+0000 32'62 54:129 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:05:55.908046+0000 0'0 2021-05-12T09:05:55.908046+0000 0 2021-05-12T09:08:28.064 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:08:28.065 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:08:28.065 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:08:28.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 2 2021-05-12T09:08:28.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:08:28.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:08:28.079 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T09:08:28.079 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:08:28.079 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T09:08:28.079 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:08:28.079 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:08:28.080 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:08:28.080 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:08:28.080 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T09:08:28.080 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T09:08:28.081 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:08:28.081 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:08:28.081 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:08:28.081 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:08:28.082 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:08:28.082 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:08:28.084 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:08:28.084 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:08:28.084 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:08:28.084 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:08:28.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:08:28.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:08:28.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:08:28.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:08:28.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:08:28.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:08:28.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:08:28.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:08:28.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:08:28.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T09:08:28.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T09:08:28.087 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:08:28.088 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:08:28.089 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T09:08:28.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T09:08:28.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:08:28.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:08:28.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:08:28.107 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:28.107+0000 7fac3b6df0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:08:28.108 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:28.109+0000 7fac3b6df0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:08:28.110 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:28.110+0000 7fac3b6df0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:08:28.493 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T09:08:28.493 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:08:28.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:08:28.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:08:28.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:08:28.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:08:28.495 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:08:28.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:08:28.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:08:28.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:08:28.626 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:28.625+0000 7fac3b6df0c0 -1 Falling back to public interface 2021-05-12T09:08:28.877 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 52 up_thru 52 down_at 35 last_clean_interval [15,32) [v2:127.0.0.1:6818/52672,v1:127.0.0.1:6819/52672] [v2:127.0.0.1:6820/52672,v1:127.0.0.1:6821/52672] exists,up 10e9d857-0855-44ba-8b8e-8d823ab9204a 2021-05-12T09:08:28.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:08:28.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:08:28.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:08:28.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 2 2021-05-12T09:08:28.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:08:28.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:08:28.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:08:28.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:08:28.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:08:28.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:08:28.882 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:08:28.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:08:28.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:08:29.298 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 52 up_thru 52 down_at 35 last_clean_interval [15,32) [v2:127.0.0.1:6818/52672,v1:127.0.0.1:6819/52672] [v2:127.0.0.1:6820/52672,v1:127.0.0.1:6821/52672] exists,up 10e9d857-0855-44ba-8b8e-8d823ab9204a 2021-05-12T09:08:29.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:08:29.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:08:29.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:08:29.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2021-05-12T09:08:31.115 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:31.113+0000 7fac3b6df0c0 -1 osd.2 54 log_to_monitors {default=true} 2021-05-12T09:08:49.301 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.2 2021-05-12T09:08:49.301 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T09:08:49.301 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T09:08:49.302 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:08:49.302 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:08:49.302 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:08:49.302 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.2.asok 2021-05-12T09:08:49.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2021-05-12T09:08:49.303 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.2.asok dump_ops_in_flight 2021-05-12T09:08:49.409 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:08:49.409 INFO:tasks.workunit.client.0.smithi028.stdout: "ops": [], 2021-05-12T09:08:49.410 INFO:tasks.workunit.client.0.smithi028.stdout: "num_ops": 0 2021-05-12T09:08:49.410 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:08:49.418 INFO:tasks.workunit.client.0.smithi028.stdout:allowing recovery 2021-05-12T09:08:49.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2021-05-12T09:08:49.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2021-05-12T09:08:49.786 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:08:49.787 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 101 0 0 0 0 48430 0 0 12 12 active+recovering 2021-05-12T09:08:44.719253+0000 54'102 59:150 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:05:59.322736+0000 0'0 2021-05-12T09:05:59.322736+0000 0 2021-05-12T09:08:49.787 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:08:32.524421+0000 32'62 59:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:05:55.908046+0000 0'0 2021-05-12T09:05:55.908046+0000 0 2021-05-12T09:08:49.787 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:08:49.788 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:08:49.788 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:08:49.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-05-12T09:08:49.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2021-05-12T09:08:49.989 INFO:tasks.workunit.client.0.smithi028.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T09:08:49.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-05-12T09:08:49.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2021-05-12T09:08:50.129 INFO:tasks.workunit.client.0.smithi028.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T09:08:50.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-05-12T09:08:50.138 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2021-05-12T09:08:50.269 INFO:tasks.workunit.client.0.smithi028.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T09:08:50.277 INFO:tasks.workunit.client.0.smithi028.stdout:reading divergent objects 2021-05-12T09:08:50.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2021-05-12T09:08:50.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2021-05-12T09:08:50.639 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:08:50.639 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 101 0 0 0 0 48430 0 0 12 12 active+recovering 2021-05-12T09:08:44.719253+0000 54'102 59:150 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:05:59.322736+0000 0'0 2021-05-12T09:05:59.322736+0000 0 2021-05-12T09:08:50.639 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:08:32.524421+0000 32'62 59:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:05:55.908046+0000 0'0 2021-05-12T09:05:55.908046+0000 0 2021-05-12T09:08:50.640 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:08:50.640 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:08:50.640 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:08:50.650 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2021-05-12T09:08:50.652 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2021-05-12T09:08:50.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:08:50.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2021-05-12T09:08:50.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:08:50.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2021-05-12T09:08:50.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:08:50.715 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2021-05-12T09:08:50.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:08:50.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2021-05-12T09:08:50.776 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2021-05-12T09:08:50.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T09:08:50.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2021-05-12T09:08:50.806 INFO:tasks.workunit.client.0.smithi028.stderr:error getting test/newobject_1: (2) No such file or directory 2021-05-12T09:08:50.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T09:08:50.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2021-05-12T09:08:50.838 INFO:tasks.workunit.client.0.smithi028.stderr:error getting test/newobject_2: (2) No such file or directory 2021-05-12T09:08:50.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2021-05-12T09:08:50.839 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-05-12T09:08:50.841 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T09:08:50.844 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 20 _merge_object_divergent_entries: merging hoid 2:70b3bf12:::existing_4:head entries: 32'103 (32'4) delete 2:70b3bf12:::existing_4:head by client.4547.0:1 2021-05-12T09:06:30.195675+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T09:08:50.845 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 20 _merge_object_divergent_entries: keeping 32'103 (32'4) delete 2:70b3bf12:::existing_4:head by client.4547.0:1 2021-05-12T09:06:30.195675+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T09:08:50.845 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T09:08:50.845 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:70b3bf12:::existing_4:head prior_version: 32'4 first_divergent_update: 32'103 last_divergent_update: 32'103 2021-05-12T09:08:50.845 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: more recent entry found: 47'101 (32'4) modify 2:70b3bf12:::existing_4:head by client.4662.0:1 2021-05-12T09:07:28.745354+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2021-05-12T09:08:50.846 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 20 _merge_object_divergent_entries: merging hoid 2:75ac5091:::newobject_2:head entries: 32'105 (0'0) modify 2:75ac5091:::newobject_2:head by client.4548.0:1 2021-05-12T09:06:30.197402+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T09:08:50.846 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 20 _merge_object_divergent_entries: keeping 32'105 (0'0) modify 2:75ac5091:::newobject_2:head by client.4548.0:1 2021-05-12T09:06:30.197402+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T09:08:50.846 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T09:08:50.846 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 32'105 last_divergent_update: 32'105 2021-05-12T09:08:50.847 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:75ac5091:::newobject_2:head has no more recent entries in log 2021-05-12T09:08:50.847 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2021-05-12T09:08:50.847 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 20 _merge_object_divergent_entries: merging hoid 2:8881e6d8:::existing_3:head entries: 32'104 (32'3) delete 2:8881e6d8:::existing_3:head by client.4546.0:1 2021-05-12T09:06:30.195362+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T09:08:50.847 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 20 _merge_object_divergent_entries: keeping 32'104 (32'3) delete 2:8881e6d8:::existing_3:head by client.4546.0:1 2021-05-12T09:06:30.195362+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T09:08:50.847 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2021-05-12T09:08:50.848 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:8881e6d8:::existing_3:head prior_version: 32'3 first_divergent_update: 32'104 last_divergent_update: 32'104 2021-05-12T09:08:50.849 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:8881e6d8:::existing_3:head has no more recent entries in log 2021-05-12T09:08:50.849 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2021-05-12T09:08:50.850 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:8881e6d8:::existing_3:head olog_can_rollback_to: 32'106 2021-05-12T09:08:50.850 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:8881e6d8:::existing_3:head cannot rollback 32'104 (32'3) delete 2:8881e6d8:::existing_3:head by client.4546.0:1 2021-05-12T09:06:30.195362+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T09:08:50.850 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2021-05-12T09:08:50.850 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:8881e6d8:::existing_3:head prior_version 32'3 <= info.log_tail 32'90 2021-05-12T09:08:50.851 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 20 _merge_object_divergent_entries: merging hoid 2:97db2a0f:::existing_2:head entries: 32'102 (32'2) modify 2:97db2a0f:::existing_2:head by client.4544.0:1 2021-05-12T09:06:30.194925+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:08:50.851 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 20 _merge_object_divergent_entries: keeping 32'102 (32'2) modify 2:97db2a0f:::existing_2:head by client.4544.0:1 2021-05-12T09:06:30.194925+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:08:50.851 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T09:08:50.851 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:97db2a0f:::existing_2:head prior_version: 32'2 first_divergent_update: 32'102 last_divergent_update: 32'102 2021-05-12T09:08:50.852 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:97db2a0f:::existing_2:head has no more recent entries in log 2021-05-12T09:08:50.852 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2021-05-12T09:08:50.852 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:97db2a0f:::existing_2:head olog_can_rollback_to: 32'106 2021-05-12T09:08:50.852 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:97db2a0f:::existing_2:head cannot rollback 32'102 (32'2) modify 2:97db2a0f:::existing_2:head by client.4544.0:1 2021-05-12T09:06:30.194925+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:08:50.853 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2021-05-12T09:08:50.853 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:97db2a0f:::existing_2:head prior_version 32'2 <= info.log_tail 32'90 2021-05-12T09:08:50.853 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 20 _merge_object_divergent_entries: merging hoid 2:d858c035:::existing_1:head entries: 32'101 (32'1) modify 2:d858c035:::existing_1:head by client.4545.0:1 2021-05-12T09:06:30.194761+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:08:50.853 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 20 _merge_object_divergent_entries: keeping 32'101 (32'1) modify 2:d858c035:::existing_1:head by client.4545.0:1 2021-05-12T09:06:30.194761+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:08:50.854 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T09:08:50.854 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:d858c035:::existing_1:head prior_version: 32'1 first_divergent_update: 32'101 last_divergent_update: 32'101 2021-05-12T09:08:50.855 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:d858c035:::existing_1:head has no more recent entries in log 2021-05-12T09:08:50.855 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2021-05-12T09:08:50.855 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:d858c035:::existing_1:head olog_can_rollback_to: 32'106 2021-05-12T09:08:50.855 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:d858c035:::existing_1:head cannot rollback 32'101 (32'1) modify 2:d858c035:::existing_1:head by client.4545.0:1 2021-05-12T09:06:30.194761+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:08:50.856 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2021-05-12T09:08:50.856 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:d858c035:::existing_1:head prior_version 32'1 <= info.log_tail 32'90 2021-05-12T09:08:50.856 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 20 _merge_object_divergent_entries: merging hoid 2:da59f6e7:::newobject_1:head entries: 32'106 (0'0) modify 2:da59f6e7:::newobject_1:head by client.4549.0:1 2021-05-12T09:06:30.198533+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T09:08:50.856 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 20 _merge_object_divergent_entries: keeping 32'106 (0'0) modify 2:da59f6e7:::newobject_1:head by client.4549.0:1 2021-05-12T09:06:30.198533+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T09:08:50.857 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T09:08:50.857 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 32'106 last_divergent_update: 32'106 2021-05-12T09:08:50.857 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:da59f6e7:::newobject_1:head has no more recent entries in log 2021-05-12T09:08:50.857 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:08:11.935+0000 7f8858445700 10 _merge_object_divergent_entries: hoid 2:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2021-05-12T09:08:50.858 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-05-12T09:08:50.858 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T09:08:50.859 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-05-12T09:08:50.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T09:08:50.859 INFO:tasks.workunit.client.0.smithi028.stdout:success 2021-05-12T09:08:50.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2021-05-12T09:08:50.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.29719.out 2021-05-12T09:08:50.861 INFO:tasks.workunit.client.0.smithi028.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.29719.out': No such file or directory 2021-05-12T09:08:50.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2021-05-12T09:08:50.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-05-12T09:08:50.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-05-12T09:08:51.355 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'test' does not exist 2021-05-12T09:08:51.364 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2021-05-12T09:08:51.365 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:08:51.365 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:08:51.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:08:51.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:08:51.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:08:51.474 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:08:51.474 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-05-12T09:08:51.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T09:08:51.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:08:51.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T09:08:51.477 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:08:51.477 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:08:51.477 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:08:51.477 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:08:51.477 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:08:51.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:08:51.479 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:08:51.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:08:51.480 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:08:51.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:08:51.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:08:51.481 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:08:51.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:08:51.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:08:51.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:08:51.484 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:08:51.484 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:08:51.485 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:08:51.485 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T09:08:51.499 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:08:51.499 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:08:51.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:08:51.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29719 2021-05-12T09:08:51.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:08:51.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:08:51.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-05-12T09:08:51.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-05-12T09:08:51.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-05-12T09:08:51.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-05-12T09:08:51.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T09:08:51.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:08:51.504 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T09:08:51.504 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:08:51.504 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:08:51.504 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:08:51.505 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:08:51.505 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:08:51.506 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:08:51.506 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:08:51.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:08:51.507 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:08:51.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:08:51.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:08:51.507 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:08:51.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:08:51.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:08:51.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:08:51.509 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:08:51.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:08:51.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:08:51.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T09:08:51.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:08:51.512 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:08:51.512 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:08:51.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29719 2021-05-12T09:08:51.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:08:51.513 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:08:51.513 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-05-12T09:08:51.515 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:08:51.515 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:08:51.515 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:08:51.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.29719 2021-05-12T09:08:51.516 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:08:51.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T09:08:51.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:08:51.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-05-12T09:08:51.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2021-05-12T09:08:51.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2021-05-12T09:08:51.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2021-05-12T09:08:51.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2021-05-12T09:08:51.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2021-05-12T09:08:51.518 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2021-05-12T09:08:51.519 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2021-05-12T09:08:51.520 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2021-05-12T09:08:51.520 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:08:51.520 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:08:51.520 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2021-05-12T09:08:51.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-05-12T09:08:51.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:08:51.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:08:51.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:08:51.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-05-12T09:08:51.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-05-12T09:08:51.877 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:08:51.877 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:08:51.877 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:08:51.878 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:08:51.878 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:08:51.878 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:08:51.878 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:08:51.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:08:52.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:08:52.124 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:08:52.124 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:08:52.124 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:08:52.124 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:08:52.125 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:08:52.125 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:08:52.125 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:08:52.125 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:08:52.126 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:08:52.126 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:08:52.126 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:08:52.126 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-mon.a.asok 2021-05-12T09:08:52.126 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:08:52.127 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29719/ceph-mon.a.asok config get fsid 2021-05-12T09:08:52.227 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:08:52.227 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:08:52.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:08:52.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:08:52.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:08:52.228 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:08:52.229 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:08:52.229 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:08:52.229 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:08:52.229 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:08:52.229 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:08:52.230 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-mon.a.asok 2021-05-12T09:08:52.230 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:08:52.230 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29719/ceph-mon.a.asok config get mon_host 2021-05-12T09:08:52.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2021-05-12T09:08:52.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-05-12T09:08:52.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:08:52.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:08:52.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:08:52.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-05-12T09:08:52.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:08:53.220 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:08:53.220 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:08:53.220 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:08:53.220 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:08:53.221 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:08:53.221 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:08:53.221 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:08:53.222 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:08:53.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:08:53.253 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-05-12T09:08:53.253 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2021-05-12T09:08:53.253 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T09:08:53.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:08:53.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:08:53.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:08:53.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-05-12T09:08:53.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:08:53.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:08:53.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:08:53.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:08:53.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T09:08:53.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T09:08:53.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:08:53.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:08:53.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:08:53.257 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:08:53.257 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:08:53.257 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:08:53.257 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:08:53.257 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:08:53.258 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:08:53.258 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:08:53.258 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:08:53.258 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:08:53.258 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:08:53.259 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:08:53.259 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:08:53.259 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:08:53.259 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:08:53.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:08:53.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:08:53.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:08:53.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:08:53.261 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/0 2021-05-12T09:08:53.261 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:08:53.261 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=18cc91da-f8e6-4da2-beb3-4002e77c4ec1 2021-05-12T09:08:53.261 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 18cc91da-f8e6-4da2-beb3-4002e77c4ec1' 2021-05-12T09:08:53.262 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 18cc91da-f8e6-4da2-beb3-4002e77c4ec1 2021-05-12T09:08:53.262 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:08:53.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAlm5tgGAhGEBAASwW/U7A9RjtX33Bn++rQrA== 2021-05-12T09:08:53.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAlm5tgGAhGEBAASwW/U7A9RjtX33Bn++rQrA=="}' 2021-05-12T09:08:53.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 18cc91da-f8e6-4da2-beb3-4002e77c4ec1 -i td/divergent-priors/0/new.json 2021-05-12T09:08:53.605 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:08:53.615 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/0/new.json 2021-05-12T09:08:53.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAlm5tgGAhGEBAASwW/U7A9RjtX33Bn++rQrA== --osd-uuid 18cc91da-f8e6-4da2-beb3-4002e77c4ec1 2021-05-12T09:08:53.648 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:53.648+0000 7fe615f1e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:08:53.650 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:53.650+0000 7fe615f1e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:08:53.651 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:53.651+0000 7fe615f1e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:08:53.651 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:53.652+0000 7fe615f1e0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T09:08:53.652 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:53.652+0000 7fe615f1e0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T09:08:53.652 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:53.652+0000 7fe615f1e0c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T09:08:53.685 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:53.686+0000 7fe615f1e0c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-05-12T09:08:57.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-05-12T09:08:57.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:08:57.930 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:08:57.930 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:08:57.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:08:58.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:08:58.187 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:08:58.187 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:08:58.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:08:58.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:08:58.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:08:58.218 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:58.218+0000 7f56614850c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:08:58.225 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:58.225+0000 7f56614850c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:08:58.226 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:58.226+0000 7f56614850c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:08:58.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:08:58.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:08:58.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:08:58.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:08:58.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:08:58.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:08:58.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:08:58.581 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:08:58.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:08:58.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:08:58.743 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:08:58.743+0000 7f56614850c0 -1 Falling back to public interface 2021-05-12T09:08:58.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:08:59.964 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:08:59.965 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:08:59.965 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:08:59.966 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:08:59.966 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:08:59.966 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:09:00.360 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:09:00.938 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:00.938+0000 7f56614850c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:09:01.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:09:01.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:09:01.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:09:01.363 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:09:01.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:09:01.364 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:09:01.761 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/56138,v1:127.0.0.1:6803/56138] [v2:127.0.0.1:6804/56138,v1:127.0.0.1:6805/56138] exists,up 18cc91da-f8e6-4da2-beb3-4002e77c4ec1 2021-05-12T09:09:01.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:09:01.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:09:01.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:09:01.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-05-12T09:09:01.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2021-05-12T09:09:01.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T09:09:01.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:09:01.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:09:01.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:09:01.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-05-12T09:09:01.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:09:01.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:09:01.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:09:01.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:09:01.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T09:09:01.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T09:09:01.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:09:01.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:09:01.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:09:01.770 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:09:01.770 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:09:01.770 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:09:01.771 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:09:01.771 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:09:01.771 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:09:01.771 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:09:01.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:09:01.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:09:01.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:09:01.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:09:01.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:09:01.773 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:09:01.773 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:09:01.773 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:09:01.773 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:09:01.773 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:09:01.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:09:01.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/1 2021-05-12T09:09:01.774 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:09:01.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=602d76d3-2336-45c7-b45a-9e2846468b99 2021-05-12T09:09:01.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 602d76d3-2336-45c7-b45a-9e2846468b99' 2021-05-12T09:09:01.775 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 602d76d3-2336-45c7-b45a-9e2846468b99 2021-05-12T09:09:01.775 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:09:01.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAtm5tg6FHELhAApodSHi46dod4I397RLdw3Q== 2021-05-12T09:09:01.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAtm5tg6FHELhAApodSHi46dod4I397RLdw3Q=="}' 2021-05-12T09:09:01.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 602d76d3-2336-45c7-b45a-9e2846468b99 -i td/divergent-priors/1/new.json 2021-05-12T09:09:02.252 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:09:02.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/1/new.json 2021-05-12T09:09:02.265 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAtm5tg6FHELhAApodSHi46dod4I397RLdw3Q== --osd-uuid 602d76d3-2336-45c7-b45a-9e2846468b99 2021-05-12T09:09:02.296 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:02.296+0000 7fc9640920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:09:02.297 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:02.298+0000 7fc9640920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:09:02.298 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:02.299+0000 7fc9640920c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:09:02.299 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:02.300+0000 7fc9640920c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T09:09:02.300 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:02.300+0000 7fc9640920c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T09:09:02.300 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:02.300+0000 7fc9640920c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T09:09:02.373 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:02.374+0000 7fc9640920c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-05-12T09:09:06.600 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-05-12T09:09:06.601 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:09:06.601 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:09:06.602 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:09:06.602 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:09:07.123 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:09:07.123 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:09:07.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:09:07.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:09:07.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:09:07.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:09:07.152 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:07.151+0000 7f22dfed30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:09:07.172 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:07.171+0000 7f22dfed30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:09:07.173 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:07.172+0000 7f22dfed30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:09:07.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:09:07.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:09:07.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:09:07.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:09:07.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:09:07.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:09:07.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:09:07.510 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:09:07.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:09:07.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:09:07.689 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:07.688+0000 7f22dfed30c0 -1 Falling back to public interface 2021-05-12T09:09:07.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:09:08.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:09:08.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:09:08.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:09:08.907 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:09:08.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:09:08.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:09:09.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:09:09.899 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:09.898+0000 7f22dfed30c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:09:10.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:09:10.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:09:10.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:09:10.301 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:09:10.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:09:10.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:09:10.699 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/57531,v1:127.0.0.1:6811/57531] [v2:127.0.0.1:6812/57531,v1:127.0.0.1:6813/57531] exists,up 602d76d3-2336-45c7-b45a-9e2846468b99 2021-05-12T09:09:10.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:09:10.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:09:10.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:09:10.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-05-12T09:09:10.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2021-05-12T09:09:10.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T09:09:10.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:09:10.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T09:09:10.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:09:10.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-05-12T09:09:10.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:09:10.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:09:10.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:09:10.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:09:10.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T09:09:10.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T09:09:10.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:09:10.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:09:10.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:09:10.714 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:09:10.714 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:09:10.715 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:09:10.715 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:09:10.715 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:09:10.715 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:09:10.716 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:09:10.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:09:10.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:09:10.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:09:10.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:09:10.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:09:10.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:09:10.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:09:10.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:09:10.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:09:10.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:09:10.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:09:10.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/2 2021-05-12T09:09:10.718 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:09:10.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=cabac835-d2f0-45a7-ba67-30ece6d00d83 2021-05-12T09:09:10.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 cabac835-d2f0-45a7-ba67-30ece6d00d83' 2021-05-12T09:09:10.719 INFO:tasks.workunit.client.0.smithi028.stdout:add osd2 cabac835-d2f0-45a7-ba67-30ece6d00d83 2021-05-12T09:09:10.720 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:09:10.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQA2m5tgssb0KhAAZf9awQhgs0m25TKj4J1VUw== 2021-05-12T09:09:10.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQA2m5tgssb0KhAAZf9awQhgs0m25TKj4J1VUw=="}' 2021-05-12T09:09:10.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new cabac835-d2f0-45a7-ba67-30ece6d00d83 -i td/divergent-priors/2/new.json 2021-05-12T09:09:11.266 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:09:11.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/2/new.json 2021-05-12T09:09:11.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA2m5tgssb0KhAAZf9awQhgs0m25TKj4J1VUw== --osd-uuid cabac835-d2f0-45a7-ba67-30ece6d00d83 2021-05-12T09:09:11.310 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:11.309+0000 7fea5d1410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:09:11.311 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:11.310+0000 7fea5d1410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:09:11.312 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:11.312+0000 7fea5d1410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:09:11.313 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:11.312+0000 7fea5d1410c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T09:09:11.313 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:11.312+0000 7fea5d1410c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T09:09:11.314 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:11.312+0000 7fea5d1410c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T09:09:11.354 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:11.353+0000 7fea5d1410c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-05-12T09:09:15.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-05-12T09:09:15.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:09:15.871 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd2 key to auth repository 2021-05-12T09:09:15.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T09:09:15.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:09:16.394 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:09:16.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T09:09:16.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:09:16.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:09:16.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:09:16.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:09:16.425 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:16.423+0000 7fe18f5780c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:09:16.426 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:16.425+0000 7fe18f5780c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:09:16.428 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:16.426+0000 7fe18f5780c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:09:16.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T09:09:16.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:09:16.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:09:16.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:09:16.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:09:16.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:09:16.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:09:16.787 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:09:16.787 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:09:16.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:09:16.942 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:16.941+0000 7fe18f5780c0 -1 Falling back to public interface 2021-05-12T09:09:17.170 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:09:18.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:09:18.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:09:18.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:09:18.174 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:09:18.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:09:18.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:09:18.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:09:19.149 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:09:19.148+0000 7fe18f5780c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T09:09:19.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:09:19.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:09:19.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:09:19.575 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:09:19.576 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:09:19.576 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:09:19.973 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/58916,v1:127.0.0.1:6819/58916] [v2:127.0.0.1:6820/58916,v1:127.0.0.1:6821/58916] exists,up cabac835-d2f0-45a7-ba67-30ece6d00d83 2021-05-12T09:09:19.973 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:09:19.974 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:09:19.974 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:09:19.974 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2021-05-12T09:09:20.781 INFO:tasks.workunit.client.0.smithi028.stderr:noout is set 2021-05-12T09:09:20.790 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2021-05-12T09:09:21.819 INFO:tasks.workunit.client.0.smithi028.stderr:noin is set 2021-05-12T09:09:21.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2021-05-12T09:09:22.757 INFO:tasks.workunit.client.0.smithi028.stderr:nodown is set 2021-05-12T09:09:22.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2021-05-12T09:09:22.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-05-12T09:09:23.373 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'test' already exists 2021-05-12T09:09:23.382 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:09:24.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2021-05-12T09:09:25.611 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 2 size to 3 2021-05-12T09:09:25.622 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2021-05-12T09:09:26.411 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 2 min_size to 2 2021-05-12T09:09:26.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2021-05-12T09:09:27.298 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 2 pg_autoscale_mode to on 2021-05-12T09:09:27.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2021-05-12T09:09:27.309 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:09:27.309 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:09:27.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:09:27.698 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:09:27.699 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:09:27.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:09:27.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:09:27.699 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:09:27.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836488 2021-05-12T09:09:27.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836488 2021-05-12T09:09:27.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488' 2021-05-12T09:09:27.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:09:27.838 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:09:27.977 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672966 2021-05-12T09:09:27.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672966 2021-05-12T09:09:27.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488 1-42949672966' 2021-05-12T09:09:27.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:09:27.979 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:09:28.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509444 2021-05-12T09:09:28.119 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509444 2021-05-12T09:09:28.119 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-64424509444' 2021-05-12T09:09:28.119 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:09:28.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836488 2021-05-12T09:09:28.120 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:09:28.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:09:28.121 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836488 2021-05-12T09:09:28.121 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:09:28.122 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836488 2021-05-12T09:09:28.122 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-05-12T09:09:28.122 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836488 2021-05-12T09:09:28.126 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:09:28.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836488 2021-05-12T09:09:28.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:09:28.536 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672966 2021-05-12T09:09:28.537 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:09:28.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:09:28.538 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672966 2021-05-12T09:09:28.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:09:28.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672966 2021-05-12T09:09:28.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-05-12T09:09:28.540 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672966 2021-05-12T09:09:28.540 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:09:28.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672966 -lt 42949672966 2021-05-12T09:09:28.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:09:28.946 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-05-12T09:09:28.947 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:09:28.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:09:28.947 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509444 2021-05-12T09:09:28.948 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:09:28.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509444 2021-05-12T09:09:28.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-05-12T09:09:28.949 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509444 2021-05-12T09:09:28.950 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:09:29.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509442 -lt 64424509444 2021-05-12T09:09:29.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:09:30.322 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:09:30.323 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:09:30.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509444 -lt 64424509444 2021-05-12T09:09:30.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2021-05-12T09:09:30.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:09:30.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:09:30.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:09:30.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:09:30.726 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T09:09:30.726 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:09:30.727 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:09:30.727 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:09:30.727 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:09:30.727 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:09:30.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:09:30.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:09:30.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:09:30.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:09:30.821 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:09:31.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:09:31.205 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:09:31.205 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:09:31.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:09:31.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:09:31.206 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:09:31.353 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836490 2021-05-12T09:09:31.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836490 2021-05-12T09:09:31.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490' 2021-05-12T09:09:31.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:09:31.354 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:09:31.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672969 2021-05-12T09:09:31.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672969 2021-05-12T09:09:31.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2021-05-12T09:09:31.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:09:31.496 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:09:31.634 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509447 2021-05-12T09:09:31.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509447 2021-05-12T09:09:31.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672969 2-64424509447' 2021-05-12T09:09:31.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:09:31.636 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-05-12T09:09:31.636 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:09:31.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:09:31.637 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836490 2021-05-12T09:09:31.637 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:09:31.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836490 2021-05-12T09:09:31.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-05-12T09:09:31.638 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836490 2021-05-12T09:09:31.639 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:09:32.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836490 2021-05-12T09:09:32.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:09:33.023 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:09:33.023 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:09:33.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836490 -lt 21474836490 2021-05-12T09:09:33.423 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:09:33.423 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672969 2021-05-12T09:09:33.424 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:09:33.424 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:09:33.425 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672969 2021-05-12T09:09:33.425 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:09:33.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672969 2021-05-12T09:09:33.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-05-12T09:09:33.428 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672969 2021-05-12T09:09:33.428 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:09:33.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672969 -lt 42949672969 2021-05-12T09:09:33.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:09:33.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509447 2021-05-12T09:09:33.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:09:33.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:09:33.837 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509447 2021-05-12T09:09:33.838 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:09:33.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509447 2021-05-12T09:09:33.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2021-05-12T09:09:33.839 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509447 2021-05-12T09:09:33.840 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:09:34.233 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509447 -lt 64424509447 2021-05-12T09:09:34.234 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:09:34.234 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:09:34.234 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:09:34.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:09:34.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:09:34.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:09:34.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:09:34.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:09:34.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:09:34.746 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:09:34.746 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:09:35.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T09:09:35.116 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:09:35.116 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:09:35.117 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:09:35.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T09:09:35.622 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:09:35.622 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:09:35.623 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2021-05-12T09:09:35.623 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2021-05-12T09:09:35.989 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:09:35.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=2 2021-05-12T09:09:35.999 INFO:tasks.workunit.client.0.smithi028.stdout:primary and soon to be divergent is 2 2021-05-12T09:09:36.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 2' 2021-05-12T09:09:36.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2021-05-12T09:09:36.374 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:09:36.375 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T09:09:24.305549+0000 0'0 32:16 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:09:36.375 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:09:20.996706+0000 33'62 33:108 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:20.116394+0000 0'0 2021-05-12T09:09:20.116394+0000 0 2021-05-12T09:09:36.375 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:09:36.375 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:09:36.377 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:09:36.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2021-05-12T09:09:36.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-05-12T09:09:36.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 2 ']' 2021-05-12T09:09:36.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2021-05-12T09:09:36.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-05-12T09:09:36.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 2 ']' 2021-05-12T09:09:36.392 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 1' 2021-05-12T09:09:36.392 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-05-12T09:09:36.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 2 ']' 2021-05-12T09:09:36.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2021-05-12T09:09:36.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2021-05-12T09:09:36.394 INFO:tasks.workunit.client.0.smithi028.stdout:writing initial objects 2021-05-12T09:09:36.395 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2021-05-12T09:09:36.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:36.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2021-05-12T09:09:36.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:36.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2021-05-12T09:09:36.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:36.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2021-05-12T09:09:36.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:36.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2021-05-12T09:09:36.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:36.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2021-05-12T09:09:37.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:37.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2021-05-12T09:09:37.122 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:37.122 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2021-05-12T09:09:37.248 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:37.248 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2021-05-12T09:09:37.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:37.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2021-05-12T09:09:37.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:37.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2021-05-12T09:09:37.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:37.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2021-05-12T09:09:37.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:37.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2021-05-12T09:09:37.850 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:37.851 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2021-05-12T09:09:37.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:37.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2021-05-12T09:09:38.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:38.069 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2021-05-12T09:09:38.211 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:38.211 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2021-05-12T09:09:38.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:38.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2021-05-12T09:09:38.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:38.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2021-05-12T09:09:38.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:38.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2021-05-12T09:09:38.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:38.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2021-05-12T09:09:38.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:38.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2021-05-12T09:09:38.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:38.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2021-05-12T09:09:39.065 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:39.066 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2021-05-12T09:09:39.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:39.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2021-05-12T09:09:39.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:39.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2021-05-12T09:09:39.459 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:39.459 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2021-05-12T09:09:39.571 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:39.572 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2021-05-12T09:09:39.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:39.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2021-05-12T09:09:39.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:39.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2021-05-12T09:09:39.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:39.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2021-05-12T09:09:40.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:40.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2021-05-12T09:09:40.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:40.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2021-05-12T09:09:40.321 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:40.322 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2021-05-12T09:09:40.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:40.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2021-05-12T09:09:40.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:40.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2021-05-12T09:09:40.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:40.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2021-05-12T09:09:40.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:40.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2021-05-12T09:09:40.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:40.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2021-05-12T09:09:41.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:41.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2021-05-12T09:09:41.142 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:41.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2021-05-12T09:09:41.251 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:41.252 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2021-05-12T09:09:41.360 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:41.360 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2021-05-12T09:09:41.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:41.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2021-05-12T09:09:41.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:41.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2021-05-12T09:09:41.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:41.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2021-05-12T09:09:41.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:41.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2021-05-12T09:09:41.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:41.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2021-05-12T09:09:42.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:42.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2021-05-12T09:09:42.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:42.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2021-05-12T09:09:42.268 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:42.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2021-05-12T09:09:42.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:42.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2021-05-12T09:09:42.566 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:42.567 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2021-05-12T09:09:42.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:42.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2021-05-12T09:09:42.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:42.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2021-05-12T09:09:42.901 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:42.902 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2021-05-12T09:09:43.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:43.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2021-05-12T09:09:43.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:43.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2021-05-12T09:09:43.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:43.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2021-05-12T09:09:43.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:43.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2021-05-12T09:09:43.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:43.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2021-05-12T09:09:43.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:43.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2021-05-12T09:09:43.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:44.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2021-05-12T09:09:44.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:44.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2021-05-12T09:09:44.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:44.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2021-05-12T09:09:44.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:44.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2021-05-12T09:09:44.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:44.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2021-05-12T09:09:44.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:44.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2021-05-12T09:09:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2021-05-12T09:09:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2021-05-12T09:09:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2021-05-12T09:09:44.737 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:44.737 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2021-05-12T09:09:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2021-05-12T09:09:44.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:44.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2021-05-12T09:09:45.088 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:45.088 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2021-05-12T09:09:45.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:45.189 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2021-05-12T09:09:45.322 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:45.322 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2021-05-12T09:09:45.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:45.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2021-05-12T09:09:45.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:45.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2021-05-12T09:09:45.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:45.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2021-05-12T09:09:45.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:45.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2021-05-12T09:09:45.934 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:45.934 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2021-05-12T09:09:46.067 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:46.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2021-05-12T09:09:46.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:46.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2021-05-12T09:09:46.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:46.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2021-05-12T09:09:46.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:46.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2021-05-12T09:09:46.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:46.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2021-05-12T09:09:46.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:46.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2021-05-12T09:09:46.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:46.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2021-05-12T09:09:46.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:46.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2021-05-12T09:09:46.981 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:46.981 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2021-05-12T09:09:47.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:47.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2021-05-12T09:09:47.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:47.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2021-05-12T09:09:47.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:47.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2021-05-12T09:09:47.425 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:47.425 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2021-05-12T09:09:47.559 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:47.559 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2021-05-12T09:09:47.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:47.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2021-05-12T09:09:47.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:47.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2021-05-12T09:09:47.902 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:47.902 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2021-05-12T09:09:48.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:48.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2021-05-12T09:09:48.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-05-12T09:09:48.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2021-05-12T09:09:48.238 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T09:09:48.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2021-05-12T09:09:48.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:09:48.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:09:48.240 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:09:48.240 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:09:48.240 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T09:09:48.240 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:09:48.240 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:09:48.241 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:09:48.241 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:09:48.241 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:09:48.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:09:48.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:09:48.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:09:48.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:09:48.298 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:09:48.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:09:48.683 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:09:48.683 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:09:48.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:09:48.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:09:48.684 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:09:48.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836496 2021-05-12T09:09:48.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836496 2021-05-12T09:09:48.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496' 2021-05-12T09:09:48.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:09:48.834 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:09:48.973 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672974 2021-05-12T09:09:48.973 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672974 2021-05-12T09:09:48.973 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496 1-42949672974' 2021-05-12T09:09:48.974 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:09:48.974 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:09:49.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509452 2021-05-12T09:09:49.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509452 2021-05-12T09:09:49.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509452' 2021-05-12T09:09:49.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:09:49.116 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2021-05-12T09:09:49.117 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:09:49.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:09:49.117 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836496 2021-05-12T09:09:49.117 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:09:49.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836496 2021-05-12T09:09:49.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-05-12T09:09:49.119 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836496 2021-05-12T09:09:49.124 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:09:49.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836494 -lt 21474836496 2021-05-12T09:09:49.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:09:50.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:09:50.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:09:50.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836496 -lt 21474836496 2021-05-12T09:09:50.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:09:50.898 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-05-12T09:09:50.898 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:09:50.900 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:09:50.901 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672974 2021-05-12T09:09:50.901 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:09:50.902 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672974 2021-05-12T09:09:50.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-05-12T09:09:50.903 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672974 2021-05-12T09:09:50.904 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:09:51.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672974 -lt 42949672974 2021-05-12T09:09:51.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:09:51.326 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2021-05-12T09:09:51.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:09:51.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:09:51.328 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509452 2021-05-12T09:09:51.328 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:09:51.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509452 2021-05-12T09:09:51.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-05-12T09:09:51.329 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509452 2021-05-12T09:09:51.330 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:09:51.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509452 -lt 64424509452 2021-05-12T09:09:51.720 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:09:51.721 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:09:51.721 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:09:52.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:09:52.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:09:52.222 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:09:52.223 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:09:52.223 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:09:52.223 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:09:52.223 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:09:52.224 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:09:52.598 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T09:09:52.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:09:52.599 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:09:52.599 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:09:53.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T09:09:53.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:09:53.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:09:53.116 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2021-05-12T09:09:53.116 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local poolname=test 2021-05-12T09:09:53.117 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: local objectname=existing_1 2021-05-12T09:09:53.117 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: ceph --format json osd map test existing_1 2021-05-12T09:09:53.117 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: jq -r .pgid 2021-05-12T09:09:53.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=2.0 2021-05-12T09:09:53.518 INFO:tasks.workunit.client.0.smithi028.stdout:blackholing osds 0 1 2021-05-12T09:09:53.519 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 1' 2021-05-12T09:09:53.519 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2021-05-12T09:09:53.884 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:09:53.884 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-05-12T09:09:24.305549+0000 33'100 33:116 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:09:53.885 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:09:20.996706+0000 33'62 33:108 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:20.116394+0000 0'0 2021-05-12T09:09:20.116394+0000 0 2021-05-12T09:09:53.885 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:09:53.885 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:09:53.886 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:09:53.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2021-05-12T09:09:53.896 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2021-05-12T09:09:53.897 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T09:09:53.897 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T09:09:53.897 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:09:53.898 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:09:53.898 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:09:53.899 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.0.asok 2021-05-12T09:09:53.899 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2021-05-12T09:09:53.900 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.0.asok config set objectstore_blackhole 1 2021-05-12T09:09:53.996 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:09:53.996 INFO:tasks.workunit.client.0.smithi028.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T09:09:53.996 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:09:53.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2021-05-12T09:09:54.000 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.1 2021-05-12T09:09:54.001 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T09:09:54.001 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T09:09:54.001 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:09:54.001 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:09:54.002 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:09:54.002 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.1.asok 2021-05-12T09:09:54.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2021-05-12T09:09:54.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.1.asok config set objectstore_blackhole 1 2021-05-12T09:09:54.096 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:09:54.097 INFO:tasks.workunit.client.0.smithi028.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T09:09:54.097 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:09:54.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2021-05-12T09:09:54.106 INFO:tasks.workunit.client.0.smithi028.stdout:create new divergent objects 2021-05-12T09:09:54.107 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2021-05-12T09:09:54.107 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T09:09:54.107 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T09:09:54.107 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2021-05-12T09:09:54.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2021-05-12T09:09:54.108 INFO:tasks.workunit.client.0.smithi028.stdout:writing divergent objects 2021-05-12T09:09:54.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2021-05-12T09:09:54.109 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2021-05-12T09:09:54.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2021-05-12T09:09:54.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2021-05-12T09:09:54.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2021-05-12T09:09:54.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2021-05-12T09:09:54.111 INFO:tasks.workunit.client.0.smithi028.stdout:remove divergent objects 2021-05-12T09:09:54.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2021-05-12T09:09:54.113 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2021-05-12T09:09:54.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-05-12T09:09:54.113 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2021-05-12T09:09:54.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2021-05-12T09:09:54.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-05-12T09:09:54.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2021-05-12T09:09:54.114 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2021-05-12T09:09:54.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2021-05-12T09:09:54.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2021-05-12T09:09:54.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2021-05-12T09:10:04.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2021-05-12T09:10:04.118 INFO:tasks.workunit.client.0.smithi028.stderr:rados: no process found 2021-05-12T09:10:04.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2021-05-12T09:10:04.119 INFO:tasks.workunit.client.0.smithi028.stdout:killing all the osds 2021-05-12T09:10:04.119 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2021-05-12T09:10:04.496 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:10:04.497 INFO:tasks.workunit.client.0.smithi028.stdout:2.3f 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.444911+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.497 INFO:tasks.workunit.client.0.smithi028.stdout:2.3e 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.443448+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.497 INFO:tasks.workunit.client.0.smithi028.stdout:2.3d 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.443868+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.497 INFO:tasks.workunit.client.0.smithi028.stdout:2.3c 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.443860+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.497 INFO:tasks.workunit.client.0.smithi028.stdout:2.3b 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.444637+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.498 INFO:tasks.workunit.client.0.smithi028.stdout:2.3a 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.444113+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.498 INFO:tasks.workunit.client.0.smithi028.stdout:2.39 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.447622+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.498 INFO:tasks.workunit.client.0.smithi028.stdout:2.38 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.445304+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.498 INFO:tasks.workunit.client.0.smithi028.stdout:2.37 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.444661+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.499 INFO:tasks.workunit.client.0.smithi028.stdout:2.36 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.443690+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.499 INFO:tasks.workunit.client.0.smithi028.stdout:2.35 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.447620+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.499 INFO:tasks.workunit.client.0.smithi028.stdout:2.34 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.447125+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.499 INFO:tasks.workunit.client.0.smithi028.stdout:2.33 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.443325+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.500 INFO:tasks.workunit.client.0.smithi028.stdout:2.32 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.443261+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.500 INFO:tasks.workunit.client.0.smithi028.stdout:2.31 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.446007+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.500 INFO:tasks.workunit.client.0.smithi028.stdout:2.30 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.446909+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.500 INFO:tasks.workunit.client.0.smithi028.stdout:2.2f 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.443729+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.501 INFO:tasks.workunit.client.0.smithi028.stdout:2.2e 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.443587+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.501 INFO:tasks.workunit.client.0.smithi028.stdout:2.2d 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.446133+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.501 INFO:tasks.workunit.client.0.smithi028.stdout:2.2c 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.445099+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.501 INFO:tasks.workunit.client.0.smithi028.stdout:2.2b 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.444437+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.502 INFO:tasks.workunit.client.0.smithi028.stdout:2.2a 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.443978+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.502 INFO:tasks.workunit.client.0.smithi028.stdout:2.29 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.446057+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.502 INFO:tasks.workunit.client.0.smithi028.stdout:2.28 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.445165+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.502 INFO:tasks.workunit.client.0.smithi028.stdout:2.27 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.443922+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.503 INFO:tasks.workunit.client.0.smithi028.stdout:2.26 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.444677+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.503 INFO:tasks.workunit.client.0.smithi028.stdout:2.25 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.444208+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.503 INFO:tasks.workunit.client.0.smithi028.stdout:2.24 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.444360+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.503 INFO:tasks.workunit.client.0.smithi028.stdout:2.23 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.443576+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.504 INFO:tasks.workunit.client.0.smithi028.stdout:2.e 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.445526+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.504 INFO:tasks.workunit.client.0.smithi028.stdout:2.d 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.446046+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.504 INFO:tasks.workunit.client.0.smithi028.stdout:2.c 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.446879+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.504 INFO:tasks.workunit.client.0.smithi028.stdout:2.b 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.446899+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.505 INFO:tasks.workunit.client.0.smithi028.stdout:2.a 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.445369+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.505 INFO:tasks.workunit.client.0.smithi028.stdout:2.9 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.444418+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.505 INFO:tasks.workunit.client.0.smithi028.stdout:2.8 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.445172+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.505 INFO:tasks.workunit.client.0.smithi028.stdout:2.7 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.444671+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.505 INFO:tasks.workunit.client.0.smithi028.stdout:2.6 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.444584+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.506 INFO:tasks.workunit.client.0.smithi028.stdout:2.5 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.445903+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.506 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 1 0 0 0 0 728 0 0 16 16 peering 2021-05-12T09:10:00.245700+0000 33'106 35:127 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.506 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:09:20.996706+0000 33'62 33:108 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:20.116394+0000 0'0 2021-05-12T09:09:20.116394+0000 0 2021-05-12T09:10:04.506 INFO:tasks.workunit.client.0.smithi028.stdout:2.3 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.445803+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.506 INFO:tasks.workunit.client.0.smithi028.stdout:2.1 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.445058+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.507 INFO:tasks.workunit.client.0.smithi028.stdout:2.2 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.445868+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.507 INFO:tasks.workunit.client.0.smithi028.stdout:2.4 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.444872+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.507 INFO:tasks.workunit.client.0.smithi028.stdout:2.f 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.446389+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.507 INFO:tasks.workunit.client.0.smithi028.stdout:2.10 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.444824+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.508 INFO:tasks.workunit.client.0.smithi028.stdout:2.11 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.444985+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.508 INFO:tasks.workunit.client.0.smithi028.stdout:2.12 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.447126+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.508 INFO:tasks.workunit.client.0.smithi028.stdout:2.13 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.446250+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.508 INFO:tasks.workunit.client.0.smithi028.stdout:2.14 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.445673+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.509 INFO:tasks.workunit.client.0.smithi028.stdout:2.15 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.447123+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.509 INFO:tasks.workunit.client.0.smithi028.stdout:2.16 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.447367+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.509 INFO:tasks.workunit.client.0.smithi028.stdout:2.17 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.447865+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.510 INFO:tasks.workunit.client.0.smithi028.stdout:2.18 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.445767+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.510 INFO:tasks.workunit.client.0.smithi028.stdout:2.19 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.444344+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.510 INFO:tasks.workunit.client.0.smithi028.stdout:2.1a 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.445785+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.510 INFO:tasks.workunit.client.0.smithi028.stdout:2.1b 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.447870+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.511 INFO:tasks.workunit.client.0.smithi028.stdout:2.1c 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.447366+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.511 INFO:tasks.workunit.client.0.smithi028.stdout:2.1d 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.446936+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.511 INFO:tasks.workunit.client.0.smithi028.stdout:2.1e 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.444100+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.511 INFO:tasks.workunit.client.0.smithi028.stdout:2.1f 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.444119+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.512 INFO:tasks.workunit.client.0.smithi028.stdout:2.20 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.445228+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.512 INFO:tasks.workunit.client.0.smithi028.stdout:2.21 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.445507+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.512 INFO:tasks.workunit.client.0.smithi028.stdout:2.22 2 0 0 0 0 729 0 0 16 16 peering 2021-05-12T09:10:00.444158+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:04.512 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:10:04.512 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:10:04.514 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:10:04.514 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2021-05-12T09:10:04.514 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:10:04.514 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:10:04.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:10:04.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:10:04.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:10:04.614 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:10:04.614 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-05-12T09:10:04.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2021-05-12T09:10:05.170 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already down. 2021-05-12T09:10:05.179 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-05-12T09:10:05.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2021-05-12T09:10:05.763 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already down. 2021-05-12T09:10:05.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-05-12T09:10:05.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2021-05-12T09:10:06.342 INFO:tasks.workunit.client.0.smithi028.stderr:osd.2 is already down. 2021-05-12T09:10:06.352 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2021-05-12T09:10:06.352 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2021-05-12T09:10:06.889 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already out. 2021-05-12T09:10:06.898 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2021-05-12T09:10:06.899 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.1 2021-05-12T09:10:07.423 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already out. 2021-05-12T09:10:07.433 INFO:tasks.workunit.client.0.smithi028.stdout:bringing up non_divergent 0 1 2021-05-12T09:10:07.433 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 1' 2021-05-12T09:10:07.433 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2021-05-12T09:10:07.803 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:10:07.803 INFO:tasks.workunit.client.0.smithi028.stdout:2.3f 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.444911+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.803 INFO:tasks.workunit.client.0.smithi028.stdout:2.3e 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.443448+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.803 INFO:tasks.workunit.client.0.smithi028.stdout:2.3d 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.443868+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.804 INFO:tasks.workunit.client.0.smithi028.stdout:2.3c 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.443860+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.804 INFO:tasks.workunit.client.0.smithi028.stdout:2.3b 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.444637+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.804 INFO:tasks.workunit.client.0.smithi028.stdout:2.3a 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.444113+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.804 INFO:tasks.workunit.client.0.smithi028.stdout:2.39 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.447622+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.805 INFO:tasks.workunit.client.0.smithi028.stdout:2.38 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.445304+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.805 INFO:tasks.workunit.client.0.smithi028.stdout:2.37 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.444661+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.805 INFO:tasks.workunit.client.0.smithi028.stdout:2.36 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.443690+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.805 INFO:tasks.workunit.client.0.smithi028.stdout:2.35 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.447620+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.805 INFO:tasks.workunit.client.0.smithi028.stdout:2.34 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.447125+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.806 INFO:tasks.workunit.client.0.smithi028.stdout:2.33 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.443325+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.806 INFO:tasks.workunit.client.0.smithi028.stdout:2.32 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.443261+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.806 INFO:tasks.workunit.client.0.smithi028.stdout:2.31 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.446007+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.806 INFO:tasks.workunit.client.0.smithi028.stdout:2.30 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.446909+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.807 INFO:tasks.workunit.client.0.smithi028.stdout:2.2f 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.443729+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.807 INFO:tasks.workunit.client.0.smithi028.stdout:2.2e 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.443587+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.807 INFO:tasks.workunit.client.0.smithi028.stdout:2.2d 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.446133+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.807 INFO:tasks.workunit.client.0.smithi028.stdout:2.2c 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.445099+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.808 INFO:tasks.workunit.client.0.smithi028.stdout:2.2b 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.444437+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.808 INFO:tasks.workunit.client.0.smithi028.stdout:2.2a 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.443978+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.808 INFO:tasks.workunit.client.0.smithi028.stdout:2.29 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.446057+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.808 INFO:tasks.workunit.client.0.smithi028.stdout:2.28 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.445165+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.808 INFO:tasks.workunit.client.0.smithi028.stdout:2.27 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.443922+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.809 INFO:tasks.workunit.client.0.smithi028.stdout:2.26 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.444677+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.809 INFO:tasks.workunit.client.0.smithi028.stdout:2.25 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.444208+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.809 INFO:tasks.workunit.client.0.smithi028.stdout:2.24 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.444360+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.809 INFO:tasks.workunit.client.0.smithi028.stdout:2.23 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.443576+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.810 INFO:tasks.workunit.client.0.smithi028.stdout:2.e 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.445526+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.810 INFO:tasks.workunit.client.0.smithi028.stdout:2.d 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.446046+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.810 INFO:tasks.workunit.client.0.smithi028.stdout:2.c 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.446879+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.810 INFO:tasks.workunit.client.0.smithi028.stdout:2.b 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.446899+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.810 INFO:tasks.workunit.client.0.smithi028.stdout:2.a 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.445369+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.811 INFO:tasks.workunit.client.0.smithi028.stdout:2.9 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.444418+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.811 INFO:tasks.workunit.client.0.smithi028.stdout:2.8 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.445172+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.811 INFO:tasks.workunit.client.0.smithi028.stdout:2.7 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.444671+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.811 INFO:tasks.workunit.client.0.smithi028.stdout:2.6 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.444584+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.812 INFO:tasks.workunit.client.0.smithi028.stdout:2.5 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.445903+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.812 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 1 0 0 0 0 728 0 0 16 16 stale+peering 2021-05-12T09:10:00.245700+0000 33'106 35:127 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.812 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 stale+active+clean 2021-05-12T09:09:20.996706+0000 33'62 33:108 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:20.116394+0000 0'0 2021-05-12T09:09:20.116394+0000 0 2021-05-12T09:10:07.812 INFO:tasks.workunit.client.0.smithi028.stdout:2.3 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.445803+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.812 INFO:tasks.workunit.client.0.smithi028.stdout:2.1 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.445058+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.813 INFO:tasks.workunit.client.0.smithi028.stdout:2.2 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.445868+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.813 INFO:tasks.workunit.client.0.smithi028.stdout:2.4 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.444872+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.813 INFO:tasks.workunit.client.0.smithi028.stdout:2.f 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.446389+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.813 INFO:tasks.workunit.client.0.smithi028.stdout:2.10 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.444824+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.814 INFO:tasks.workunit.client.0.smithi028.stdout:2.11 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.444985+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.814 INFO:tasks.workunit.client.0.smithi028.stdout:2.12 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.447126+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.814 INFO:tasks.workunit.client.0.smithi028.stdout:2.13 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.446250+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.814 INFO:tasks.workunit.client.0.smithi028.stdout:2.14 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.445673+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.814 INFO:tasks.workunit.client.0.smithi028.stdout:2.15 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.447123+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.815 INFO:tasks.workunit.client.0.smithi028.stdout:2.16 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.447367+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.815 INFO:tasks.workunit.client.0.smithi028.stdout:2.17 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.447865+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.815 INFO:tasks.workunit.client.0.smithi028.stdout:2.18 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.445767+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.815 INFO:tasks.workunit.client.0.smithi028.stdout:2.19 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.444344+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.816 INFO:tasks.workunit.client.0.smithi028.stdout:2.1a 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.445785+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.816 INFO:tasks.workunit.client.0.smithi028.stdout:2.1b 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.447870+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.816 INFO:tasks.workunit.client.0.smithi028.stdout:2.1c 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.447366+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.816 INFO:tasks.workunit.client.0.smithi028.stdout:2.1d 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.446936+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.816 INFO:tasks.workunit.client.0.smithi028.stdout:2.1e 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.444100+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.817 INFO:tasks.workunit.client.0.smithi028.stdout:2.1f 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.444119+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.817 INFO:tasks.workunit.client.0.smithi028.stdout:2.20 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.445228+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.817 INFO:tasks.workunit.client.0.smithi028.stdout:2.21 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.445507+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.817 INFO:tasks.workunit.client.0.smithi028.stdout:2.22 2 0 0 0 0 729 0 0 16 16 stale+peering 2021-05-12T09:10:00.444158+0000 33'106 35:125 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:07.818 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:10:07.818 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:10:07.822 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:10:07.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2021-05-12T09:10:07.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2021-05-12T09:10:07.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:10:07.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:10:07.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T09:10:07.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:10:07.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T09:10:07.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:10:07.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:10:07.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:10:07.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:10:07.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T09:10:07.825 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T09:10:07.825 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:10:07.825 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:10:07.825 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:10:07.825 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:10:07.826 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:10:07.826 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:10:07.826 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:10:07.826 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:10:07.826 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:10:07.827 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:10:07.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:10:07.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:10:07.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:10:07.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:10:07.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:10:07.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:10:07.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:10:07.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:10:07.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:10:07.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T09:10:07.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T09:10:07.830 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:10:07.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:10:07.831 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T09:10:07.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T09:10:07.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:10:07.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:10:07.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:10:07.845 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:10:07.845+0000 7f28151c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:10:07.847 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:10:07.847+0000 7f28151c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:10:07.848 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:10:07.848+0000 7f28151c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:10:08.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T09:10:08.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:10:08.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:10:08.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:10:08.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:10:08.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:10:08.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:10:08.203 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:10:08.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:10:08.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:10:08.371 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:10:08.371+0000 7f28151c50c0 -1 Falling back to public interface 2021-05-12T09:10:08.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:10:09.622 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:10:09.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:10:09.623 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:10:09.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:10:09.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:10:09.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:10:10.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:10:11.018 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:10:11.019 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:10:11.019 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:10:11.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:10:11.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:10:11.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:10:11.051 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:10:11.051+0000 7f28151c50c0 -1 osd.0 33 log_to_monitors {default=true} 2021-05-12T09:10:11.432 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up out weight 0 up_from 43 up_thru 0 down_at 37 last_clean_interval [5,33) [v2:127.0.0.1:6802/63042,v1:127.0.0.1:6803/63042] [v2:127.0.0.1:6804/63042,v1:127.0.0.1:6805/63042] exists,up 18cc91da-f8e6-4da2-beb3-4002e77c4ec1 2021-05-12T09:10:11.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:10:11.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:10:11.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:10:11.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2021-05-12T09:10:11.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 1 2021-05-12T09:10:11.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:10:11.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:10:11.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T09:10:11.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:10:11.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T09:10:11.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:10:11.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:10:11.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:10:11.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:10:11.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T09:10:11.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T09:10:11.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:10:11.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:10:11.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:10:11.441 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:10:11.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:10:11.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:10:11.442 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:10:11.442 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:10:11.442 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:10:11.443 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:10:11.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:10:11.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:10:11.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:10:11.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:10:11.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:10:11.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:10:11.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:10:11.445 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:10:11.445 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:10:11.445 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T09:10:11.445 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T09:10:11.446 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:10:11.447 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:10:11.447 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T09:10:11.448 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T09:10:11.448 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:10:11.448 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:10:11.448 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:10:11.470 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:10:11.470+0000 7f804f6ca0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:10:11.472 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:10:11.472+0000 7f804f6ca0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:10:11.475 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:10:11.474+0000 7f804f6ca0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:10:11.845 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T09:10:11.845 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:10:11.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:10:11.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:10:11.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:10:11.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:10:11.847 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:10:11.847 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:10:11.848 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:10:11.848 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:10:11.994 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:10:11.994+0000 7f804f6ca0c0 -1 Falling back to public interface 2021-05-12T09:10:12.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:10:13.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:10:13.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:10:13.274 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:10:13.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:10:13.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:10:13.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:10:13.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:10:14.577 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:10:14.576+0000 7f804f6ca0c0 -1 osd.1 33 log_to_monitors {default=true} 2021-05-12T09:10:14.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:10:14.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:10:14.675 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:10:14.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:10:14.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:10:14.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:10:15.089 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up out weight 0 up_from 45 up_thru 16 down_at 38 last_clean_interval [10,33) [v2:127.0.0.1:6810/63599,v1:127.0.0.1:6811/63599] [v2:127.0.0.1:6812/63599,v1:127.0.0.1:6813/63599] exists,up 602d76d3-2336-45c7-b45a-9e2846468b99 2021-05-12T09:10:15.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:10:15.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:10:15.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:10:15.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2021-05-12T09:10:15.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2021-05-12T09:10:15.617 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already in. 2021-05-12T09:10:15.626 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2021-05-12T09:10:15.627 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.1 2021-05-12T09:10:16.501 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already in. 2021-05-12T09:10:16.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T09:10:16.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2021-05-12T09:10:16.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:10:16.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:10:16.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:10:16.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:10:16.512 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T09:10:16.512 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:10:16.512 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:10:16.513 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:10:16.513 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:10:16.514 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:10:16.569 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:10:16.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:10:16.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:10:16.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:10:16.570 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:10:17.029 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:10:17.029 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:10:17.030 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:10:17.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:10:17.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:10:17.031 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:10:17.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=184683593732 2021-05-12T09:10:17.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 184683593732 2021-05-12T09:10:17.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-184683593732' 2021-05-12T09:10:17.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:10:17.175 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:10:17.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=193273528323 2021-05-12T09:10:17.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 193273528323 2021-05-12T09:10:17.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-184683593732 1-193273528323' 2021-05-12T09:10:17.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:10:17.316 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:10:17.424 INFO:tasks.workunit.client.0.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-05-12T09:10:17.425 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T09:10:17.425 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T09:10:17.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T09:10:17.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:10:17.427 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-184683593732 2021-05-12T09:10:17.427 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:10:17.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:10:17.428 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-184683593732 2021-05-12T09:10:17.428 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:10:17.429 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=184683593732 2021-05-12T09:10:17.429 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 184683593732 2021-05-12T09:10:17.430 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 184683593732' 2021-05-12T09:10:17.430 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:10:17.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 1 -lt 184683593732 2021-05-12T09:10:17.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:10:18.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:10:18.814 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:10:19.208 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 184683593732 -lt 184683593732 2021-05-12T09:10:19.209 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:10:19.210 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-193273528323 2021-05-12T09:10:19.210 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:10:19.210 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:10:19.211 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-193273528323 2021-05-12T09:10:19.212 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:10:19.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=193273528323 2021-05-12T09:10:19.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 193273528323' 2021-05-12T09:10:19.212 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 193273528323 2021-05-12T09:10:19.213 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:10:19.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 193273528323 -lt 193273528323 2021-05-12T09:10:19.611 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:10:19.611 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:10:19.611 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:10:20.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 65 == 0 2021-05-12T09:10:20.141 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:10:20.141 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:10:20.141 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:10:20.141 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:10:20.141 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:10:20.142 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:10:20.142 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:10:20.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:10:20.524 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:10:20.524 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:10:20.525 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:10:21.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:10:21.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T09:10:21.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T09:10:21.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T09:10:21.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:10:21.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:10:21.135 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:10:21.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:10:21.136 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:10:21.136 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:10:21.136 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:10:21.137 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:10:21.137 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:10:21.137 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:10:21.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:10:21.525 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:10:21.525 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:10:21.526 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:10:22.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:10:22.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:10:22.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:10:22.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:10:22.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:10:22.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:10:22.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:10:22.034 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:10:22.034 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:10:22.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:10:22.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:10:22.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T09:10:22.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:10:22.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T09:10:22.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:10:22.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:10:22.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:10:22.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:10:22.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:10:22.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:10:22.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:10:22.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:10:23.130 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:10:23.131 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:10:23.131 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:10:23.132 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:10:23.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:10:23.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:10:23.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:10:23.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:10:23.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:10:23.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:10:23.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:10:23.646 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:10:23.647 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:10:24.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:10:24.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:10:24.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T09:10:24.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:10:24.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T09:10:24.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:10:24.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:10:24.553 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:10:24.553 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:10:24.554 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:10:24.554 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:10:24.554 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:10:24.554 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:10:24.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:10:24.943 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:10:24.943 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:10:24.943 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:10:25.452 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:10:25.452 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:10:25.452 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:10:25.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:10:25.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:10:25.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:10:25.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:10:25.453 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:10:25.454 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:10:25.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:10:25.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:10:25.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T09:10:25.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:10:25.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T09:10:26.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:10:26.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:10:26.755 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:10:26.755 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:10:26.755 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:10:26.755 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:10:26.756 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:10:26.756 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:10:27.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:10:27.153 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:10:27.154 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:10:27.154 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:10:27.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:10:27.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:10:27.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:10:27.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:10:27.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:10:27.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:10:27.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:10:27.692 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:10:27.693 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:10:28.195 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:10:28.195 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:10:28.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T09:10:28.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:10:28.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T09:10:29.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:10:29.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:10:29.797 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:10:29.798 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:10:29.798 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:10:29.798 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:10:29.798 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:10:29.799 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:10:30.194 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:10:30.195 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:10:30.195 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:10:30.196 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:10:30.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:10:30.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:10:30.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:10:30.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:10:30.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:10:30.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:10:30.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:10:30.711 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:10:30.711 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:10:31.215 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:10:31.215 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:10:31.215 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T09:10:31.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:10:31.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T09:10:34.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:10:34.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:10:34.419 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:10:34.419 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:10:34.419 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:10:34.421 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:10:34.421 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:10:34.421 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:10:34.817 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:10:34.817 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:10:34.817 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:10:34.818 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:10:35.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:10:35.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:10:35.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:10:35.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:10:35.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:10:35.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:10:35.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:10:35.326 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:10:35.326 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:10:35.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:10:35.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:10:35.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T09:10:35.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:10:35.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T09:10:42.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:10:42.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:10:42.230 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:10:42.231 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:10:42.231 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:10:42.231 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:10:42.232 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:10:42.232 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:10:42.629 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:10:42.630 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:10:42.630 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:10:42.631 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:10:43.134 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:10:43.135 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:10:43.135 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:10:43.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:10:43.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:10:43.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:10:43.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:10:43.137 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:10:43.137 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:10:43.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:10:43.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:10:43.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T09:10:43.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:10:43.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T09:10:50.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:10:50.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:10:50.941 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:10:50.941 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:10:50.941 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:10:50.941 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:10:50.942 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:10:50.942 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:10:51.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:10:51.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:10:51.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:10:51.338 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:10:51.845 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:10:51.845 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:10:51.845 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:10:51.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:10:51.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:10:51.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:10:51.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:10:51.847 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:10:51.847 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:10:52.346 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:10:52.346 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:10:52.346 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T09:10:52.346 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T09:10:52.851 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:10:52.852 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_fingerprint": "69070f6e-49a7-4aa1-b600-c76826add343", 2021-05-12T09:10:52.852 INFO:tasks.workunit.client.0.smithi028.stdout: "version": "17.0.0-4095-ga8c673d5", 2021-05-12T09:10:52.852 INFO:tasks.workunit.client.0.smithi028.stdout: "commit": "a8c673d555488beec9ccda5812ba822e69c12ea9", 2021-05-12T09:10:52.852 INFO:tasks.workunit.client.0.smithi028.stdout: "timestamp": "2021-05-12T09:10:52.849796+0000", 2021-05-12T09:10:52.852 INFO:tasks.workunit.client.0.smithi028.stdout: "tag": "", 2021-05-12T09:10:52.853 INFO:tasks.workunit.client.0.smithi028.stdout: "health": { 2021-05-12T09:10:52.853 INFO:tasks.workunit.client.0.smithi028.stdout: "status": "HEALTH_WARN", 2021-05-12T09:10:52.853 INFO:tasks.workunit.client.0.smithi028.stdout: "checks": { 2021-05-12T09:10:52.853 INFO:tasks.workunit.client.0.smithi028.stdout: "OSDMAP_FLAGS": { 2021-05-12T09:10:52.854 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:10:52.854 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:10:52.854 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T09:10:52.854 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 17 2021-05-12T09:10:52.854 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.855 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [], 2021-05-12T09:10:52.855 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:10:52.855 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.855 INFO:tasks.workunit.client.0.smithi028.stdout: "OSD_DOWN": { 2021-05-12T09:10:52.855 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:10:52.856 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:10:52.856 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "1 osds down", 2021-05-12T09:10:52.856 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 1 2021-05-12T09:10:52.856 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.857 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:10:52.857 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.857 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "osd.2 (root=default,host=smithi028) is down" 2021-05-12T09:10:52.857 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.857 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:52.858 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:10:52.858 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.858 INFO:tasks.workunit.client.0.smithi028.stdout: "PG_DEGRADED": { 2021-05-12T09:10:52.858 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:10:52.858 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:10:52.859 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "Degraded data redundancy: 102/306 objects degraded (33.333%), 65 pgs degraded", 2021-05-12T09:10:52.859 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 65 2021-05-12T09:10:52.859 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.859 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:10:52.859 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.860 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.860 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.860 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.861 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.0 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.861 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.861 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.861 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.1 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.862 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.862 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.862 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.2 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.863 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.863 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.863 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.3 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.864 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.864 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.864 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.4 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.864 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.864 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.865 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.5 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.865 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.865 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.865 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.6 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.865 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.866 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.866 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.7 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.866 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.866 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.867 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.8 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.867 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.867 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.867 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.9 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.867 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.868 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.868 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.a is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.868 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.868 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.869 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.b is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.869 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.869 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.869 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.c is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.869 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.870 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.870 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.d is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.870 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.870 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.871 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.e is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.871 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.871 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.871 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.f is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.871 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.872 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.872 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.10 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.872 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.873 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.873 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.11 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.873 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.874 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.874 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.12 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.874 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.874 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.874 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.13 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.875 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.875 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.875 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.14 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.875 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.875 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.876 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.23 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.876 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.876 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.876 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.24 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.877 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.877 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.877 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.25 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.877 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.878 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.878 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.26 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.878 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.878 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.879 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.27 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.879 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.879 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.879 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.28 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.879 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.880 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.880 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.29 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.880 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.880 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.880 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.2a is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.881 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.881 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.881 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.2b is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.881 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.882 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.882 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.2c is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.882 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.882 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.882 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.2d is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.883 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.883 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.883 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.2e is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.883 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.883 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.884 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.2f is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.884 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.884 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.884 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.30 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.884 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.885 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.885 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.31 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.885 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.886 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.886 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.32 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.886 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.886 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.886 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.33 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.887 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.887 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.887 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.34 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.887 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.887 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.888 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.35 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.888 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.888 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.888 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.36 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.888 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.889 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.889 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.37 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.889 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.889 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.890 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.38 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.890 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.890 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.890 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.39 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.890 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.891 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.891 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.3a is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.891 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.891 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.892 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.3b is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.892 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.892 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.893 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.3c is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.893 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.893 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.893 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.3d is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.894 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.894 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.894 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.3e is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.894 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.895 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.895 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.3f is active+undersized+degraded, acting [1,0]" 2021-05-12T09:10:52.895 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.895 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:52.895 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:10:52.896 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.896 INFO:tasks.workunit.client.0.smithi028.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T09:10:52.896 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:10:52.896 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:10:52.897 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T09:10:52.897 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 1 2021-05-12T09:10:52.897 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.897 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:10:52.897 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.898 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T09:10:52.898 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.898 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.899 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T09:10:52.899 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.899 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:52.899 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:10:52.899 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.900 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.900 INFO:tasks.workunit.client.0.smithi028.stdout: "mutes": [] 2021-05-12T09:10:52.900 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.900 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap_first_committed": 1, 2021-05-12T09:10:52.900 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap_last_committed": 1, 2021-05-12T09:10:52.901 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap": { 2021-05-12T09:10:52.901 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 1, 2021-05-12T09:10:52.901 INFO:tasks.workunit.client.0.smithi028.stdout: "fsid": "8342ec9e-531f-44b4-8c8c-079d667fe908", 2021-05-12T09:10:52.901 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:08:51.540323Z", 2021-05-12T09:10:52.902 INFO:tasks.workunit.client.0.smithi028.stdout: "created": "2021-05-12T09:08:51.540323Z", 2021-05-12T09:10:52.902 INFO:tasks.workunit.client.0.smithi028.stdout: "min_mon_release": 17, 2021-05-12T09:10:52.902 INFO:tasks.workunit.client.0.smithi028.stdout: "min_mon_release_name": "quincy", 2021-05-12T09:10:52.902 INFO:tasks.workunit.client.0.smithi028.stdout: "election_strategy": 1, 2021-05-12T09:10:52.903 INFO:tasks.workunit.client.0.smithi028.stdout: "disallowed_leaders: ": "", 2021-05-12T09:10:52.903 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode": false, 2021-05-12T09:10:52.903 INFO:tasks.workunit.client.0.smithi028.stdout: "features": { 2021-05-12T09:10:52.903 INFO:tasks.workunit.client.0.smithi028.stdout: "persistent": [ 2021-05-12T09:10:52.903 INFO:tasks.workunit.client.0.smithi028.stdout: "kraken", 2021-05-12T09:10:52.904 INFO:tasks.workunit.client.0.smithi028.stdout: "luminous", 2021-05-12T09:10:52.904 INFO:tasks.workunit.client.0.smithi028.stdout: "mimic", 2021-05-12T09:10:52.904 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap-prune", 2021-05-12T09:10:52.904 INFO:tasks.workunit.client.0.smithi028.stdout: "nautilus", 2021-05-12T09:10:52.904 INFO:tasks.workunit.client.0.smithi028.stdout: "octopus", 2021-05-12T09:10:52.905 INFO:tasks.workunit.client.0.smithi028.stdout: "pacific", 2021-05-12T09:10:52.905 INFO:tasks.workunit.client.0.smithi028.stdout: "elector-pinging", 2021-05-12T09:10:52.905 INFO:tasks.workunit.client.0.smithi028.stdout: "quincy" 2021-05-12T09:10:52.905 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:52.905 INFO:tasks.workunit.client.0.smithi028.stdout: "optional": [] 2021-05-12T09:10:52.906 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.906 INFO:tasks.workunit.client.0.smithi028.stdout: "mons": [ 2021-05-12T09:10:52.906 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.906 INFO:tasks.workunit.client.0.smithi028.stdout: "rank": 0, 2021-05-12T09:10:52.907 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "a", 2021-05-12T09:10:52.907 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:10:52.907 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:10:52.907 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.907 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:10:52.908 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:7115", 2021-05-12T09:10:52.908 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 0 2021-05-12T09:10:52.908 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.908 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.908 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.909 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T09:10:52.909 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T09:10:52.909 INFO:tasks.workunit.client.0.smithi028.stdout: "priority": 0, 2021-05-12T09:10:52.909 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 0, 2021-05-12T09:10:52.910 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_location": "{}" 2021-05-12T09:10:52.910 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.910 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.910 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.911 INFO:tasks.workunit.client.0.smithi028.stdout: "quorum": [ 2021-05-12T09:10:52.911 INFO:tasks.workunit.client.0.smithi028.stdout: 0 2021-05-12T09:10:52.911 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:52.911 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap": { 2021-05-12T09:10:52.911 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 51, 2021-05-12T09:10:52.912 INFO:tasks.workunit.client.0.smithi028.stdout: "fsid": "8342ec9e-531f-44b4-8c8c-079d667fe908", 2021-05-12T09:10:52.912 INFO:tasks.workunit.client.0.smithi028.stdout: "created": "2021-05-12T09:08:52.384986+0000", 2021-05-12T09:10:52.912 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:10:16.883217+0000", 2021-05-12T09:10:52.912 INFO:tasks.workunit.client.0.smithi028.stdout: "last_up_change": "2021-05-12T09:10:14.790190+0000", 2021-05-12T09:10:52.912 INFO:tasks.workunit.client.0.smithi028.stdout: "last_in_change": "2021-05-12T09:10:16.072476+0000", 2021-05-12T09:10:52.913 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T09:10:52.913 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_num": 5800384, 2021-05-12T09:10:52.913 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_set": [ 2021-05-12T09:10:52.913 INFO:tasks.workunit.client.0.smithi028.stdout: "nodown", 2021-05-12T09:10:52.914 INFO:tasks.workunit.client.0.smithi028.stdout: "noin", 2021-05-12T09:10:52.914 INFO:tasks.workunit.client.0.smithi028.stdout: "noout", 2021-05-12T09:10:52.914 INFO:tasks.workunit.client.0.smithi028.stdout: "pglog_hardlimit", 2021-05-12T09:10:52.914 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snapdirs", 2021-05-12T09:10:52.914 INFO:tasks.workunit.client.0.smithi028.stdout: "recovery_deletes", 2021-05-12T09:10:52.915 INFO:tasks.workunit.client.0.smithi028.stdout: "sortbitwise" 2021-05-12T09:10:52.915 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:52.915 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_version": 9, 2021-05-12T09:10:52.915 INFO:tasks.workunit.client.0.smithi028.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T09:10:52.915 INFO:tasks.workunit.client.0.smithi028.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T09:10:52.916 INFO:tasks.workunit.client.0.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T09:10:52.916 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_snapshot": "", 2021-05-12T09:10:52.916 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_max": 2, 2021-05-12T09:10:52.916 INFO:tasks.workunit.client.0.smithi028.stdout: "max_osd": 3, 2021-05-12T09:10:52.917 INFO:tasks.workunit.client.0.smithi028.stdout: "require_min_compat_client": "luminous", 2021-05-12T09:10:52.917 INFO:tasks.workunit.client.0.smithi028.stdout: "min_compat_client": "jewel", 2021-05-12T09:10:52.917 INFO:tasks.workunit.client.0.smithi028.stdout: "require_osd_release": "quincy", 2021-05-12T09:10:52.917 INFO:tasks.workunit.client.0.smithi028.stdout: "pools": [ 2021-05-12T09:10:52.917 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.918 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 1, 2021-05-12T09:10:52.918 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_name": ".mgr", 2021-05-12T09:10:52.918 INFO:tasks.workunit.client.0.smithi028.stdout: "create_time": "2021-05-12T09:09:20.066263+0000", 2021-05-12T09:10:52.918 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": 1, 2021-05-12T09:10:52.918 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_names": "hashpspool", 2021-05-12T09:10:52.919 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:10:52.919 INFO:tasks.workunit.client.0.smithi028.stdout: "size": 3, 2021-05-12T09:10:52.919 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 2, 2021-05-12T09:10:52.919 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_rule": 0, 2021-05-12T09:10:52.919 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_count": 0, 2021-05-12T09:10:52.920 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_target": 0, 2021-05-12T09:10:52.920 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T09:10:52.920 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T09:10:52.920 INFO:tasks.workunit.client.0.smithi028.stdout: "object_hash": 2, 2021-05-12T09:10:52.921 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_autoscale_mode": "off", 2021-05-12T09:10:52.921 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num": 1, 2021-05-12T09:10:52.921 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num": 1, 2021-05-12T09:10:52.921 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num_target": 1, 2021-05-12T09:10:52.921 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_target": 1, 2021-05-12T09:10:52.922 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_pending": 1, 2021-05-12T09:10:52.922 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pg_merge_meta": { 2021-05-12T09:10:52.922 INFO:tasks.workunit.client.0.smithi028.stdout: "source_pgid": "0.0", 2021-05-12T09:10:52.922 INFO:tasks.workunit.client.0.smithi028.stdout: "ready_epoch": 0, 2021-05-12T09:10:52.923 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_started": 0, 2021-05-12T09:10:52.923 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": 0, 2021-05-12T09:10:52.923 INFO:tasks.workunit.client.0.smithi028.stdout: "source_version": "0'0", 2021-05-12T09:10:52.923 INFO:tasks.workunit.client.0.smithi028.stdout: "target_version": "0'0" 2021-05-12T09:10:52.923 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.924 INFO:tasks.workunit.client.0.smithi028.stdout: "last_change": "20", 2021-05-12T09:10:52.924 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend": "0", 2021-05-12T09:10:52.924 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T09:10:52.924 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T09:10:52.924 INFO:tasks.workunit.client.0.smithi028.stdout: "auid": 0, 2021-05-12T09:10:52.925 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_mode": "selfmanaged", 2021-05-12T09:10:52.925 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_seq": 0, 2021-05-12T09:10:52.925 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_epoch": 0, 2021-05-12T09:10:52.925 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_snaps": [], 2021-05-12T09:10:52.926 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps": "[]", 2021-05-12T09:10:52.926 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_bytes": 0, 2021-05-12T09:10:52.926 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_objects": 0, 2021-05-12T09:10:52.926 INFO:tasks.workunit.client.0.smithi028.stdout: "tiers": [], 2021-05-12T09:10:52.926 INFO:tasks.workunit.client.0.smithi028.stdout: "tier_of": -1, 2021-05-12T09:10:52.927 INFO:tasks.workunit.client.0.smithi028.stdout: "read_tier": -1, 2021-05-12T09:10:52.927 INFO:tasks.workunit.client.0.smithi028.stdout: "write_tier": -1, 2021-05-12T09:10:52.927 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_mode": "none", 2021-05-12T09:10:52.927 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_bytes": 0, 2021-05-12T09:10:52.928 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_objects": 0, 2021-05-12T09:10:52.928 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T09:10:52.928 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T09:10:52.928 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T09:10:52.928 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_flush_age": 0, 2021-05-12T09:10:52.929 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_evict_age": 0, 2021-05-12T09:10:52.929 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profile": "", 2021-05-12T09:10:52.929 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_params": { 2021-05-12T09:10:52.929 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "none" 2021-05-12T09:10:52.929 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.930 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_period": 0, 2021-05-12T09:10:52.930 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_count": 0, 2021-05-12T09:10:52.930 INFO:tasks.workunit.client.0.smithi028.stdout: "use_gmt_hitset": true, 2021-05-12T09:10:52.930 INFO:tasks.workunit.client.0.smithi028.stdout: "min_read_recency_for_promote": 0, 2021-05-12T09:10:52.930 INFO:tasks.workunit.client.0.smithi028.stdout: "min_write_recency_for_promote": 0, 2021-05-12T09:10:52.931 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T09:10:52.931 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_search_last_n": 0, 2021-05-12T09:10:52.931 INFO:tasks.workunit.client.0.smithi028.stdout: "grade_table": [], 2021-05-12T09:10:52.931 INFO:tasks.workunit.client.0.smithi028.stdout: "stripe_width": 0, 2021-05-12T09:10:52.931 INFO:tasks.workunit.client.0.smithi028.stdout: "expected_num_objects": 0, 2021-05-12T09:10:52.932 INFO:tasks.workunit.client.0.smithi028.stdout: "fast_read": false, 2021-05-12T09:10:52.932 INFO:tasks.workunit.client.0.smithi028.stdout: "options": { 2021-05-12T09:10:52.932 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_min": 1 2021-05-12T09:10:52.932 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.932 INFO:tasks.workunit.client.0.smithi028.stdout: "application_metadata": { 2021-05-12T09:10:52.933 INFO:tasks.workunit.client.0.smithi028.stdout: "mgr": {} 2021-05-12T09:10:52.933 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.933 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.933 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.934 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 2, 2021-05-12T09:10:52.934 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_name": "test", 2021-05-12T09:10:52.934 INFO:tasks.workunit.client.0.smithi028.stdout: "create_time": "2021-05-12T09:09:23.059340+0000", 2021-05-12T09:10:52.934 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": 1, 2021-05-12T09:10:52.934 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_names": "hashpspool", 2021-05-12T09:10:52.935 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:10:52.935 INFO:tasks.workunit.client.0.smithi028.stdout: "size": 3, 2021-05-12T09:10:52.935 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 2, 2021-05-12T09:10:52.935 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_rule": 0, 2021-05-12T09:10:52.935 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_count": 0, 2021-05-12T09:10:52.936 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_target": 0, 2021-05-12T09:10:52.936 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T09:10:52.936 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T09:10:52.936 INFO:tasks.workunit.client.0.smithi028.stdout: "object_hash": 2, 2021-05-12T09:10:52.937 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_autoscale_mode": "on", 2021-05-12T09:10:52.937 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num": 64, 2021-05-12T09:10:52.937 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num": 1, 2021-05-12T09:10:52.937 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num_target": 64, 2021-05-12T09:10:52.937 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_target": 64, 2021-05-12T09:10:52.938 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_pending": 64, 2021-05-12T09:10:52.938 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pg_merge_meta": { 2021-05-12T09:10:52.938 INFO:tasks.workunit.client.0.smithi028.stdout: "source_pgid": "0.0", 2021-05-12T09:10:52.938 INFO:tasks.workunit.client.0.smithi028.stdout: "ready_epoch": 0, 2021-05-12T09:10:52.938 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_started": 0, 2021-05-12T09:10:52.939 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": 0, 2021-05-12T09:10:52.939 INFO:tasks.workunit.client.0.smithi028.stdout: "source_version": "0'0", 2021-05-12T09:10:52.939 INFO:tasks.workunit.client.0.smithi028.stdout: "target_version": "0'0" 2021-05-12T09:10:52.939 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.939 INFO:tasks.workunit.client.0.smithi028.stdout: "last_change": "35", 2021-05-12T09:10:52.940 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend": "0", 2021-05-12T09:10:52.940 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T09:10:52.940 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_preluminous": "35", 2021-05-12T09:10:52.940 INFO:tasks.workunit.client.0.smithi028.stdout: "auid": 0, 2021-05-12T09:10:52.941 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_mode": "selfmanaged", 2021-05-12T09:10:52.941 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_seq": 0, 2021-05-12T09:10:52.941 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_epoch": 0, 2021-05-12T09:10:52.941 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_snaps": [], 2021-05-12T09:10:52.941 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps": "[]", 2021-05-12T09:10:52.942 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_bytes": 0, 2021-05-12T09:10:52.942 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_objects": 0, 2021-05-12T09:10:52.942 INFO:tasks.workunit.client.0.smithi028.stdout: "tiers": [], 2021-05-12T09:10:52.942 INFO:tasks.workunit.client.0.smithi028.stdout: "tier_of": -1, 2021-05-12T09:10:52.942 INFO:tasks.workunit.client.0.smithi028.stdout: "read_tier": -1, 2021-05-12T09:10:52.943 INFO:tasks.workunit.client.0.smithi028.stdout: "write_tier": -1, 2021-05-12T09:10:52.943 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_mode": "none", 2021-05-12T09:10:52.943 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_bytes": 0, 2021-05-12T09:10:52.943 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_objects": 0, 2021-05-12T09:10:52.944 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T09:10:52.944 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T09:10:52.944 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T09:10:52.944 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_flush_age": 0, 2021-05-12T09:10:52.945 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_evict_age": 0, 2021-05-12T09:10:52.945 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profile": "", 2021-05-12T09:10:52.945 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_params": { 2021-05-12T09:10:52.945 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "none" 2021-05-12T09:10:52.945 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.946 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_period": 0, 2021-05-12T09:10:52.946 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_count": 0, 2021-05-12T09:10:52.946 INFO:tasks.workunit.client.0.smithi028.stdout: "use_gmt_hitset": true, 2021-05-12T09:10:52.946 INFO:tasks.workunit.client.0.smithi028.stdout: "min_read_recency_for_promote": 0, 2021-05-12T09:10:52.946 INFO:tasks.workunit.client.0.smithi028.stdout: "min_write_recency_for_promote": 0, 2021-05-12T09:10:52.947 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T09:10:52.947 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_search_last_n": 0, 2021-05-12T09:10:52.947 INFO:tasks.workunit.client.0.smithi028.stdout: "grade_table": [], 2021-05-12T09:10:52.947 INFO:tasks.workunit.client.0.smithi028.stdout: "stripe_width": 0, 2021-05-12T09:10:52.948 INFO:tasks.workunit.client.0.smithi028.stdout: "expected_num_objects": 0, 2021-05-12T09:10:52.948 INFO:tasks.workunit.client.0.smithi028.stdout: "fast_read": false, 2021-05-12T09:10:52.948 INFO:tasks.workunit.client.0.smithi028.stdout: "options": {}, 2021-05-12T09:10:52.948 INFO:tasks.workunit.client.0.smithi028.stdout: "application_metadata": {} 2021-05-12T09:10:52.948 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.949 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:52.949 INFO:tasks.workunit.client.0.smithi028.stdout: "osds": [ 2021-05-12T09:10:52.949 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.949 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:10:52.949 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "18cc91da-f8e6-4da2-beb3-4002e77c4ec1", 2021-05-12T09:10:52.950 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 1, 2021-05-12T09:10:52.950 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:10:52.950 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:10:52.950 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:10:52.950 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 5, 2021-05-12T09:10:52.951 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 33, 2021-05-12T09:10:52.951 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 43, 2021-05-12T09:10:52.951 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 48, 2021-05-12T09:10:52.951 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 37, 2021-05-12T09:10:52.951 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:10:52.952 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:10:52.952 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:10:52.952 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.952 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:10:52.953 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6802", 2021-05-12T09:10:52.953 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:10:52.953 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.953 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.953 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:10:52.954 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6803", 2021-05-12T09:10:52.954 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:10:52.954 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.954 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.954 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.955 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:10:52.955 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:10:52.955 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.955 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:10:52.956 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6804", 2021-05-12T09:10:52.956 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:10:52.956 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.956 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.956 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:10:52.957 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6805", 2021-05-12T09:10:52.957 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:10:52.957 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.957 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.957 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.958 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:10:52.958 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:10:52.958 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.958 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:10:52.959 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6808", 2021-05-12T09:10:52.959 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:10:52.959 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.959 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.959 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:10:52.960 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6809", 2021-05-12T09:10:52.960 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:10:52.960 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.960 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.961 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.961 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:10:52.961 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:10:52.961 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.961 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:10:52.962 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6806", 2021-05-12T09:10:52.962 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:10:52.962 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.962 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.962 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:10:52.963 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6807", 2021-05-12T09:10:52.963 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:10:52.963 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.964 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.964 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.964 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6803/63042", 2021-05-12T09:10:52.964 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6805/63042", 2021-05-12T09:10:52.964 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6809/63042", 2021-05-12T09:10:52.965 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6807/63042", 2021-05-12T09:10:52.965 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:10:52.965 INFO:tasks.workunit.client.0.smithi028.stdout: "exists", 2021-05-12T09:10:52.965 INFO:tasks.workunit.client.0.smithi028.stdout: "up" 2021-05-12T09:10:52.965 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.966 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.966 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.966 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:10:52.966 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "602d76d3-2336-45c7-b45a-9e2846468b99", 2021-05-12T09:10:52.967 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 1, 2021-05-12T09:10:52.967 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:10:52.967 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:10:52.967 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:10:52.967 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 10, 2021-05-12T09:10:52.968 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 33, 2021-05-12T09:10:52.968 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 45, 2021-05-12T09:10:52.968 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 50, 2021-05-12T09:10:52.968 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 38, 2021-05-12T09:10:52.968 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:10:52.969 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:10:52.969 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:10:52.969 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.969 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:10:52.969 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6810", 2021-05-12T09:10:52.970 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:10:52.970 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.970 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.970 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:10:52.970 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6811", 2021-05-12T09:10:52.971 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:10:52.971 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.971 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.971 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.971 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:10:52.972 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:10:52.972 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.972 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:10:52.973 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6812", 2021-05-12T09:10:52.973 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:10:52.973 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.973 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.973 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:10:52.974 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6813", 2021-05-12T09:10:52.974 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:10:52.974 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.974 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.974 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.975 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:10:52.975 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:10:52.975 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.975 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:10:52.976 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6816", 2021-05-12T09:10:52.976 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:10:52.976 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.976 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.976 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:10:52.977 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6817", 2021-05-12T09:10:52.977 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:10:52.977 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.977 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.977 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.978 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:10:52.978 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:10:52.978 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.978 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:10:52.979 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6814", 2021-05-12T09:10:52.979 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:10:52.979 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.979 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.979 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:10:52.980 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6815", 2021-05-12T09:10:52.980 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:10:52.980 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.980 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.980 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.981 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6811/63599", 2021-05-12T09:10:52.981 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6813/63599", 2021-05-12T09:10:52.981 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6817/63599", 2021-05-12T09:10:52.981 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6815/63599", 2021-05-12T09:10:52.981 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:10:52.982 INFO:tasks.workunit.client.0.smithi028.stdout: "exists", 2021-05-12T09:10:52.982 INFO:tasks.workunit.client.0.smithi028.stdout: "up" 2021-05-12T09:10:52.982 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.982 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.982 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.983 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:10:52.983 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "cabac835-d2f0-45a7-ba67-30ece6d00d83", 2021-05-12T09:10:52.983 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 0, 2021-05-12T09:10:52.983 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:10:52.984 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:10:52.984 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:10:52.984 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 0, 2021-05-12T09:10:52.984 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 0, 2021-05-12T09:10:52.984 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 15, 2021-05-12T09:10:52.985 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 35, 2021-05-12T09:10:52.985 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 39, 2021-05-12T09:10:52.985 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:10:52.985 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:10:52.985 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:10:52.986 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.986 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:10:52.986 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6818", 2021-05-12T09:10:52.986 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:10:52.986 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.987 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.987 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:10:52.987 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6819", 2021-05-12T09:10:52.987 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:10:52.987 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.988 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.988 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.988 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:10:52.988 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:10:52.988 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.989 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:10:52.989 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6820", 2021-05-12T09:10:52.989 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:10:52.989 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.990 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.990 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:10:52.990 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6821", 2021-05-12T09:10:52.990 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:10:52.990 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.991 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.991 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.991 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:10:52.991 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:10:52.991 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.992 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:10:52.992 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6824", 2021-05-12T09:10:52.992 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:10:52.992 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.993 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.993 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:10:52.993 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6825", 2021-05-12T09:10:52.993 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:10:52.994 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.994 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.994 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.995 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:10:52.995 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:10:52.995 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.995 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:10:52.995 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6822", 2021-05-12T09:10:52.996 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:10:52.996 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.996 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:52.996 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:10:52.996 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6823", 2021-05-12T09:10:52.997 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:10:52.997 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.997 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.997 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:52.997 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6819/58916", 2021-05-12T09:10:52.998 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6821/58916", 2021-05-12T09:10:52.998 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6825/58916", 2021-05-12T09:10:52.998 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6823/58916", 2021-05-12T09:10:52.998 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:10:52.999 INFO:tasks.workunit.client.0.smithi028.stdout: "exists" 2021-05-12T09:10:52.999 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:52.999 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:52.999 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:52.999 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_xinfo": [ 2021-05-12T09:10:53.000 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.000 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:10:53.000 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:10:04.942621+0000", 2021-05-12T09:10:53.000 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:10:53.000 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:10:53.001 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:10:53.001 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:10:53.001 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:09:01.894764+0000", 2021-05-12T09:10:53.001 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:10:53.001 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.002 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.002 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:10:53.002 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:10:05.572036+0000", 2021-05-12T09:10:53.002 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:10:53.002 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:10:53.003 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:10:53.003 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:10:53.003 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:09:10.922242+0000", 2021-05-12T09:10:53.003 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:10:53.003 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.004 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.004 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:10:53.004 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:10:06.119786+0000", 2021-05-12T09:10:53.004 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:10:53.005 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:10:53.005 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:10:53.005 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:10:53.005 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:09:20.187472+0000", 2021-05-12T09:10:53.005 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:10:53.006 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.006 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:53.006 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_upmap": [], 2021-05-12T09:10:53.006 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_upmap_items": [], 2021-05-12T09:10:53.006 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_temp": [], 2021-05-12T09:10:53.007 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_temp": [], 2021-05-12T09:10:53.007 INFO:tasks.workunit.client.0.smithi028.stdout: "blocklist": {}, 2021-05-12T09:10:53.007 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profiles": { 2021-05-12T09:10:53.007 INFO:tasks.workunit.client.0.smithi028.stdout: "default": { 2021-05-12T09:10:53.007 INFO:tasks.workunit.client.0.smithi028.stdout: "k": "2", 2021-05-12T09:10:53.008 INFO:tasks.workunit.client.0.smithi028.stdout: "m": "2", 2021-05-12T09:10:53.008 INFO:tasks.workunit.client.0.smithi028.stdout: "plugin": "jerasure", 2021-05-12T09:10:53.008 INFO:tasks.workunit.client.0.smithi028.stdout: "technique": "reed_sol_van" 2021-05-12T09:10:53.008 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.008 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.009 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps_queue": [], 2021-05-12T09:10:53.009 INFO:tasks.workunit.client.0.smithi028.stdout: "new_removed_snaps": [], 2021-05-12T09:10:53.009 INFO:tasks.workunit.client.0.smithi028.stdout: "new_purged_snaps": [], 2021-05-12T09:10:53.009 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_node_flags": {}, 2021-05-12T09:10:53.010 INFO:tasks.workunit.client.0.smithi028.stdout: "device_class_flags": {}, 2021-05-12T09:10:53.010 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode": { 2021-05-12T09:10:53.010 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode_enabled": false, 2021-05-12T09:10:53.010 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_bucket_count": 0, 2021-05-12T09:10:53.010 INFO:tasks.workunit.client.0.smithi028.stdout: "degraded_stretch_mode": 0, 2021-05-12T09:10:53.011 INFO:tasks.workunit.client.0.smithi028.stdout: "recovering_stretch_mode": 0, 2021-05-12T09:10:53.011 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode_bucket": 0 2021-05-12T09:10:53.011 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.011 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.011 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_metadata": [ 2021-05-12T09:10:53.012 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.012 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:10:53.012 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:10:53.012 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6804/63042,v1:127.0.0.1:6805/63042]", 2021-05-12T09:10:53.012 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:10:53.013 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:10:53.013 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:10:53.013 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:10:53.013 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:10:53.013 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:10:53.014 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:10:53.014 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:10:53.014 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:10:53.014 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T09:10:53.014 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:10:53.015 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:10:53.015 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:10:53.015 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:10:53.015 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:10:53.015 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:10:53.016 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:10:53.016 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:10:53.016 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:10:53.016 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:10:53.017 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:10:53.017 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:10:53.017 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:10:53.017 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:10:53.017 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:10:53.018 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6802/63042,v1:127.0.0.1:6803/63042]", 2021-05-12T09:10:53.018 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:10:53.018 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/63042,v1:127.0.0.1:6809/63042]", 2021-05-12T09:10:53.018 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/63042,v1:127.0.0.1:6807/63042]", 2021-05-12T09:10:53.018 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:10:53.019 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:10:53.019 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:10:53.019 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:10:53.019 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:10:53.019 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:10:53.020 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:10:53.020 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:10:53.020 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:10:53.020 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T09:10:53.020 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:10:53.021 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:10:53.021 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:10:53.021 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.021 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.022 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:10:53.022 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:10:53.022 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6812/63599,v1:127.0.0.1:6813/63599]", 2021-05-12T09:10:53.022 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:10:53.022 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:10:53.023 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:10:53.023 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:10:53.023 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:10:53.023 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:10:53.024 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:10:53.024 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:10:53.024 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:10:53.024 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T09:10:53.024 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:10:53.025 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:10:53.025 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:10:53.025 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:10:53.025 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:10:53.025 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:10:53.026 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:10:53.026 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:10:53.026 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:10:53.026 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:10:53.026 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:10:53.027 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:10:53.027 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:10:53.027 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:10:53.027 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:10:53.028 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6810/63599,v1:127.0.0.1:6811/63599]", 2021-05-12T09:10:53.028 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:10:53.028 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/63599,v1:127.0.0.1:6817/63599]", 2021-05-12T09:10:53.028 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/63599,v1:127.0.0.1:6815/63599]", 2021-05-12T09:10:53.029 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:10:53.029 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:10:53.029 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:10:53.029 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:10:53.029 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:10:53.030 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:10:53.030 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:10:53.030 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:10:53.030 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:10:53.030 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T09:10:53.031 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:10:53.031 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:10:53.031 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:10:53.031 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.032 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.032 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:10:53.032 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:10:53.032 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6820/58916,v1:127.0.0.1:6821/58916]", 2021-05-12T09:10:53.032 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:10:53.033 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:10:53.033 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:10:53.033 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:10:53.033 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:10:53.034 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:10:53.034 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:10:53.034 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:10:53.034 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:10:53.034 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T09:10:53.035 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:10:53.035 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:10:53.035 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:10:53.036 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:10:53.036 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:10:53.036 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:10:53.036 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:10:53.037 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:10:53.037 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:10:53.037 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:10:53.038 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:10:53.038 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:10:53.038 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:10:53.038 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:10:53.039 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:10:53.039 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6818/58916,v1:127.0.0.1:6819/58916]", 2021-05-12T09:10:53.039 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:10:53.039 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/58916,v1:127.0.0.1:6825/58916]", 2021-05-12T09:10:53.040 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/58916,v1:127.0.0.1:6823/58916]", 2021-05-12T09:10:53.040 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:10:53.040 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:10:53.040 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:10:53.041 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:10:53.041 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:10:53.041 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:10:53.041 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:10:53.042 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:10:53.042 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:10:53.042 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T09:10:53.043 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:10:53.043 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:10:53.043 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:10:53.043 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.044 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:53.044 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_clean_epochs": { 2021-05-12T09:10:53.044 INFO:tasks.workunit.client.0.smithi028.stdout: "min_last_epoch_clean": 0, 2021-05-12T09:10:53.045 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": { 2021-05-12T09:10:53.045 INFO:tasks.workunit.client.0.smithi028.stdout: "per_pool": [] 2021-05-12T09:10:53.045 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.045 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_epochs": [ 2021-05-12T09:10:53.046 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.046 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:10:53.046 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 43 2021-05-12T09:10:53.046 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.047 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.047 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:10:53.047 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 46 2021-05-12T09:10:53.047 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.048 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.048 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:10:53.048 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 15 2021-05-12T09:10:53.048 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.048 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:53.049 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.049 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_first_committed": 1, 2021-05-12T09:10:53.049 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_last_committed": 51, 2021-05-12T09:10:53.049 INFO:tasks.workunit.client.0.smithi028.stdout: "crushmap": { 2021-05-12T09:10:53.049 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": [ 2021-05-12T09:10:53.050 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.050 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:10:53.050 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.0", 2021-05-12T09:10:53.050 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:10:53.051 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.051 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.051 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:10:53.051 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.1", 2021-05-12T09:10:53.051 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:10:53.052 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.052 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.052 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:10:53.052 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.2", 2021-05-12T09:10:53.053 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:10:53.053 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.053 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:53.053 INFO:tasks.workunit.client.0.smithi028.stdout: "types": [ 2021-05-12T09:10:53.054 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.054 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 0, 2021-05-12T09:10:53.054 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd" 2021-05-12T09:10:53.054 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.055 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.055 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:10:53.055 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "host" 2021-05-12T09:10:53.055 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.056 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.056 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 2, 2021-05-12T09:10:53.056 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "chassis" 2021-05-12T09:10:53.056 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.057 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.057 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 3, 2021-05-12T09:10:53.057 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "rack" 2021-05-12T09:10:53.057 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.058 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.058 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 4, 2021-05-12T09:10:53.058 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "row" 2021-05-12T09:10:53.059 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.059 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.059 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 5, 2021-05-12T09:10:53.059 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "pdu" 2021-05-12T09:10:53.060 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.060 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.060 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 6, 2021-05-12T09:10:53.060 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "pod" 2021-05-12T09:10:53.061 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.061 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.061 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 7, 2021-05-12T09:10:53.061 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "room" 2021-05-12T09:10:53.062 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.062 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.062 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 8, 2021-05-12T09:10:53.062 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "datacenter" 2021-05-12T09:10:53.063 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.063 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.064 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 9, 2021-05-12T09:10:53.064 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "zone" 2021-05-12T09:10:53.064 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.064 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.065 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 10, 2021-05-12T09:10:53.065 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "region" 2021-05-12T09:10:53.065 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.065 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.065 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:10:53.066 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "root" 2021-05-12T09:10:53.066 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.066 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:53.066 INFO:tasks.workunit.client.0.smithi028.stdout: "buckets": [ 2021-05-12T09:10:53.067 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.067 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -1, 2021-05-12T09:10:53.067 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "default", 2021-05-12T09:10:53.067 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:10:53.067 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "root", 2021-05-12T09:10:53.068 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:10:53.068 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:10:53.068 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:10:53.068 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:10:53.069 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.069 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -3, 2021-05-12T09:10:53.069 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:10:53.069 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:10:53.069 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.070 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:53.070 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.070 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.070 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -2, 2021-05-12T09:10:53.071 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "default~hdd", 2021-05-12T09:10:53.071 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:10:53.071 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "root", 2021-05-12T09:10:53.071 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:10:53.071 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:10:53.072 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:10:53.072 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:10:53.072 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.072 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -4, 2021-05-12T09:10:53.072 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:10:53.073 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:10:53.073 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.073 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:53.074 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.074 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.074 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -3, 2021-05-12T09:10:53.074 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "smithi028", 2021-05-12T09:10:53.074 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:10:53.075 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "host", 2021-05-12T09:10:53.075 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:10:53.075 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:10:53.075 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:10:53.075 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:10:53.076 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.076 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:10:53.076 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:10:53.076 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:10:53.077 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.077 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.077 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:10:53.077 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:10:53.078 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 1 2021-05-12T09:10:53.078 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.078 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.078 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:10:53.078 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:10:53.079 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 2 2021-05-12T09:10:53.079 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.079 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:53.079 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.079 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.080 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -4, 2021-05-12T09:10:53.080 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "smithi028~hdd", 2021-05-12T09:10:53.080 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:10:53.080 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "host", 2021-05-12T09:10:53.081 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:10:53.081 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:10:53.081 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:10:53.081 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:10:53.081 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.082 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:10:53.082 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:10:53.082 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:10:53.082 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.083 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.083 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:10:53.083 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:10:53.083 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 1 2021-05-12T09:10:53.083 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.084 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.084 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:10:53.084 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:10:53.084 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 2 2021-05-12T09:10:53.084 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.085 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:53.085 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.085 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:53.085 INFO:tasks.workunit.client.0.smithi028.stdout: "rules": [ 2021-05-12T09:10:53.085 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.086 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_id": 0, 2021-05-12T09:10:53.086 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_name": "replicated_rule", 2021-05-12T09:10:53.086 INFO:tasks.workunit.client.0.smithi028.stdout: "ruleset": 0, 2021-05-12T09:10:53.086 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:10:53.087 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 1, 2021-05-12T09:10:53.087 INFO:tasks.workunit.client.0.smithi028.stdout: "max_size": 10, 2021-05-12T09:10:53.087 INFO:tasks.workunit.client.0.smithi028.stdout: "steps": [ 2021-05-12T09:10:53.087 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.087 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "take", 2021-05-12T09:10:53.088 INFO:tasks.workunit.client.0.smithi028.stdout: "item": -1, 2021-05-12T09:10:53.088 INFO:tasks.workunit.client.0.smithi028.stdout: "item_name": "default" 2021-05-12T09:10:53.088 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.088 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.088 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "choose_firstn", 2021-05-12T09:10:53.089 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 0, 2021-05-12T09:10:53.089 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "osd" 2021-05-12T09:10:53.089 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.089 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.089 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "emit" 2021-05-12T09:10:53.090 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.090 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:10:53.090 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.090 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:53.090 INFO:tasks.workunit.client.0.smithi028.stdout: "tunables": { 2021-05-12T09:10:53.091 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_local_tries": 0, 2021-05-12T09:10:53.091 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_local_fallback_tries": 0, 2021-05-12T09:10:53.091 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_total_tries": 50, 2021-05-12T09:10:53.091 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_descend_once": 1, 2021-05-12T09:10:53.092 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_vary_r": 1, 2021-05-12T09:10:53.092 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_stable": 1, 2021-05-12T09:10:53.092 INFO:tasks.workunit.client.0.smithi028.stdout: "straw_calc_version": 1, 2021-05-12T09:10:53.092 INFO:tasks.workunit.client.0.smithi028.stdout: "allowed_bucket_algs": 54, 2021-05-12T09:10:53.092 INFO:tasks.workunit.client.0.smithi028.stdout: "profile": "jewel", 2021-05-12T09:10:53.093 INFO:tasks.workunit.client.0.smithi028.stdout: "optimal_tunables": 1, 2021-05-12T09:10:53.093 INFO:tasks.workunit.client.0.smithi028.stdout: "legacy_tunables": 0, 2021-05-12T09:10:53.093 INFO:tasks.workunit.client.0.smithi028.stdout: "minimum_required_version": "jewel", 2021-05-12T09:10:53.093 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables": 1, 2021-05-12T09:10:53.094 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables2": 1, 2021-05-12T09:10:53.094 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v2_rules": 0, 2021-05-12T09:10:53.094 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables3": 1, 2021-05-12T09:10:53.095 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v3_rules": 0, 2021-05-12T09:10:53.095 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v4_buckets": 1, 2021-05-12T09:10:53.095 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables5": 1, 2021-05-12T09:10:53.095 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v5_rules": 0 2021-05-12T09:10:53.095 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.096 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_args": {} 2021-05-12T09:10:53.096 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.096 INFO:tasks.workunit.client.0.smithi028.stdout: "fsmap": { 2021-05-12T09:10:53.096 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 1, 2021-05-12T09:10:53.096 INFO:tasks.workunit.client.0.smithi028.stdout: "default_fscid": -1, 2021-05-12T09:10:53.097 INFO:tasks.workunit.client.0.smithi028.stdout: "compat": { 2021-05-12T09:10:53.097 INFO:tasks.workunit.client.0.smithi028.stdout: "compat": {}, 2021-05-12T09:10:53.097 INFO:tasks.workunit.client.0.smithi028.stdout: "ro_compat": {}, 2021-05-12T09:10:53.097 INFO:tasks.workunit.client.0.smithi028.stdout: "incompat": { 2021-05-12T09:10:53.097 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_1": "base v0.20", 2021-05-12T09:10:53.098 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_2": "client writeable ranges", 2021-05-12T09:10:53.098 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T09:10:53.098 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_4": "dir inode in separate object", 2021-05-12T09:10:53.098 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T09:10:53.098 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T09:10:53.099 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_8": "no anchor table", 2021-05-12T09:10:53.099 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_9": "file layout v2", 2021-05-12T09:10:53.099 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_10": "snaprealm v2" 2021-05-12T09:10:53.099 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.099 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.100 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_flags": { 2021-05-12T09:10:53.100 INFO:tasks.workunit.client.0.smithi028.stdout: "enable_multiple": true, 2021-05-12T09:10:53.100 INFO:tasks.workunit.client.0.smithi028.stdout: "ever_enabled_multiple": true 2021-05-12T09:10:53.100 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.100 INFO:tasks.workunit.client.0.smithi028.stdout: "standbys": [], 2021-05-12T09:10:53.101 INFO:tasks.workunit.client.0.smithi028.stdout: "filesystems": [] 2021-05-12T09:10:53.101 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.101 INFO:tasks.workunit.client.0.smithi028.stdout: "mdsmap_first_committed": 1, 2021-05-12T09:10:53.101 INFO:tasks.workunit.client.0.smithi028.stdout: "mdsmap_last_committed": 1, 2021-05-12T09:10:53.101 INFO:tasks.workunit.client.0.smithi028.stdout: "auth": { 2021-05-12T09:10:53.102 INFO:tasks.workunit.client.0.smithi028.stdout: "first_committed": 1, 2021-05-12T09:10:53.102 INFO:tasks.workunit.client.0.smithi028.stdout: "last_committed": 5, 2021-05-12T09:10:53.102 INFO:tasks.workunit.client.0.smithi028.stdout: "num_secrets": 3 2021-05-12T09:10:53.102 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.103 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 65, 2021-05-12T09:10:53.103 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_active": 65, 2021-05-12T09:10:53.103 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_unknown": 0, 2021-05-12T09:10:53.103 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osd": 3, 2021-05-12T09:10:53.103 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_sum": { 2021-05-12T09:10:53.104 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:10:53.104 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 507580, 2021-05-12T09:10:53.104 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 102, 2021-05-12T09:10:53.104 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:10:53.104 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 306, 2021-05-12T09:10:53.105 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:10:53.105 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:10:53.105 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 102, 2021-05-12T09:10:53.105 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:10:53.105 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:10:53.106 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 102, 2021-05-12T09:10:53.106 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:10:53.106 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 76, 2021-05-12T09:10:53.106 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 64, 2021-05-12T09:10:53.106 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 195, 2021-05-12T09:10:53.107 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 1210, 2021-05-12T09:10:53.107 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:10:53.107 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:10:53.107 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:10:53.108 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:10:53.108 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:10:53.108 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:10:53.108 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:10:53.108 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:10:53.109 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:10:53.109 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:10:53.109 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:10:53.109 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:10:53.109 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:10:53.110 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:10:53.110 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:10:53.110 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:10:53.110 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:10:53.110 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:10:53.111 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:10:53.111 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:10:53.111 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:10:53.111 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:10:53.111 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:10:53.112 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:10:53.112 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:10:53.112 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.112 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:10:53.112 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:10:53.113 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:10:53.113 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:10:53.113 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 0, 2021-05-12T09:10:53.113 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 0, 2021-05-12T09:10:53.114 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:10:53.114 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:10:53.114 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:10:53.114 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:10:53.114 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:10:53.115 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.115 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 1038, 2021-05-12T09:10:53.115 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 1038, 2021-05-12T09:10:53.115 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 130, 2021-05-12T09:10:53.115 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 130, 2021-05-12T09:10:53.116 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 0 2021-05-12T09:10:53.116 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.116 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_sum": { 2021-05-12T09:10:53.116 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 0, 2021-05-12T09:10:53.116 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 0, 2021-05-12T09:10:53.117 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pgs": 132, 2021-05-12T09:10:53.117 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osds": 3, 2021-05-12T09:10:53.117 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_osds": 3, 2021-05-12T09:10:53.117 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T09:10:53.117 INFO:tasks.workunit.client.0.smithi028.stdout: "kb": 314572800, 2021-05-12T09:10:53.118 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used": 24652, 2021-05-12T09:10:53.118 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_data": 3700, 2021-05-12T09:10:53.118 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_omap": 2, 2021-05-12T09:10:53.118 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_meta": 20925, 2021-05-12T09:10:53.118 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_avail": 314548148, 2021-05-12T09:10:53.119 INFO:tasks.workunit.client.0.smithi028.stdout: "statfs": { 2021-05-12T09:10:53.119 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 322122547200, 2021-05-12T09:10:53.119 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 322097303552, 2021-05-12T09:10:53.119 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:10:53.120 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 3788800, 2021-05-12T09:10:53.120 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1930253, 2021-05-12T09:10:53.120 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:10:53.120 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:10:53.120 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:10:53.121 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 2160, 2021-05-12T09:10:53.121 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 21428112 2021-05-12T09:10:53.121 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.121 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_peers": [], 2021-05-12T09:10:53.121 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_trim_queue_len": 0, 2021-05-12T09:10:53.122 INFO:tasks.workunit.client.0.smithi028.stdout: "num_snap_trimming": 0, 2021-05-12T09:10:53.122 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shards_repaired": 0, 2021-05-12T09:10:53.122 INFO:tasks.workunit.client.0.smithi028.stdout: "op_queue_age_hist": { 2021-05-12T09:10:53.122 INFO:tasks.workunit.client.0.smithi028.stdout: "histogram": [], 2021-05-12T09:10:53.122 INFO:tasks.workunit.client.0.smithi028.stdout: "upper_bound": 1 2021-05-12T09:10:53.123 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.123 INFO:tasks.workunit.client.0.smithi028.stdout: "perf_stat": { 2021-05-12T09:10:53.123 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ms": 56, 2021-05-12T09:10:53.123 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ms": 56, 2021-05-12T09:10:53.123 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ns": 56000000, 2021-05-12T09:10:53.124 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ns": 56000000 2021-05-12T09:10:53.124 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.124 INFO:tasks.workunit.client.0.smithi028.stdout: "alerts": [], 2021-05-12T09:10:53.124 INFO:tasks.workunit.client.0.smithi028.stdout: "network_ping_times": [] 2021-05-12T09:10:53.124 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.125 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_sum_by_class": { 2021-05-12T09:10:53.125 INFO:tasks.workunit.client.0.smithi028.stdout: "hdd": { 2021-05-12T09:10:53.125 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 0, 2021-05-12T09:10:53.125 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 0, 2021-05-12T09:10:53.125 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pgs": 132, 2021-05-12T09:10:53.126 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osds": 3, 2021-05-12T09:10:53.126 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_osds": 3, 2021-05-12T09:10:53.126 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T09:10:53.126 INFO:tasks.workunit.client.0.smithi028.stdout: "kb": 314572800, 2021-05-12T09:10:53.127 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used": 24652, 2021-05-12T09:10:53.127 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_data": 3700, 2021-05-12T09:10:53.127 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_omap": 2, 2021-05-12T09:10:53.127 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_meta": 20925, 2021-05-12T09:10:53.127 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_avail": 314548148, 2021-05-12T09:10:53.128 INFO:tasks.workunit.client.0.smithi028.stdout: "statfs": { 2021-05-12T09:10:53.128 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 322122547200, 2021-05-12T09:10:53.128 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 322097303552, 2021-05-12T09:10:53.128 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:10:53.128 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 3788800, 2021-05-12T09:10:53.129 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1930253, 2021-05-12T09:10:53.129 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:10:53.129 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:10:53.129 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:10:53.130 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 2160, 2021-05-12T09:10:53.130 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 21428112 2021-05-12T09:10:53.130 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.130 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_peers": [], 2021-05-12T09:10:53.130 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_trim_queue_len": 0, 2021-05-12T09:10:53.131 INFO:tasks.workunit.client.0.smithi028.stdout: "num_snap_trimming": 0, 2021-05-12T09:10:53.131 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shards_repaired": 0, 2021-05-12T09:10:53.131 INFO:tasks.workunit.client.0.smithi028.stdout: "op_queue_age_hist": { 2021-05-12T09:10:53.131 INFO:tasks.workunit.client.0.smithi028.stdout: "histogram": [], 2021-05-12T09:10:53.132 INFO:tasks.workunit.client.0.smithi028.stdout: "upper_bound": 1 2021-05-12T09:10:53.132 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.132 INFO:tasks.workunit.client.0.smithi028.stdout: "perf_stat": { 2021-05-12T09:10:53.132 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ms": 56, 2021-05-12T09:10:53.132 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ms": 56, 2021-05-12T09:10:53.133 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ns": 56000000, 2021-05-12T09:10:53.133 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ns": 56000000 2021-05-12T09:10:53.133 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.133 INFO:tasks.workunit.client.0.smithi028.stdout: "alerts": [], 2021-05-12T09:10:53.134 INFO:tasks.workunit.client.0.smithi028.stdout: "network_ping_times": [] 2021-05-12T09:10:53.134 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.134 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.134 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_stats": [ 2021-05-12T09:10:53.134 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.135 INFO:tasks.workunit.client.0.smithi028.stdout: "poolid": 2, 2021-05-12T09:10:53.135 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 64, 2021-05-12T09:10:53.135 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:10:53.135 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 48300, 2021-05-12T09:10:53.135 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 100, 2021-05-12T09:10:53.136 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:10:53.136 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 300, 2021-05-12T09:10:53.136 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:10:53.136 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:10:53.136 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 100, 2021-05-12T09:10:53.137 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:10:53.137 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:10:53.137 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 100, 2021-05-12T09:10:53.137 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:10:53.137 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 0, 2021-05-12T09:10:53.138 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 0, 2021-05-12T09:10:53.138 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 100, 2021-05-12T09:10:53.138 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 100, 2021-05-12T09:10:53.138 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:10:53.138 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:10:53.139 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:10:53.139 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:10:53.139 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:10:53.139 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:10:53.139 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:10:53.140 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:10:53.140 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:10:53.140 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:10:53.140 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:10:53.141 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:10:53.141 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:10:53.141 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:10:53.141 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:10:53.141 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:10:53.142 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:10:53.142 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:10:53.142 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:10:53.142 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:10:53.142 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:10:53.143 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:10:53.143 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:10:53.143 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:10:53.143 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:10:53.143 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.144 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:10:53.144 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:10:53.144 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:10:53.144 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:10:53.145 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 1220608, 2021-05-12T09:10:53.145 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 143228, 2021-05-12T09:10:53.145 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:10:53.145 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:10:53.145 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:10:53.146 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:10:53.146 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:10:53.146 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.146 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 1024, 2021-05-12T09:10:53.147 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 1024, 2021-05-12T09:10:53.147 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 128, 2021-05-12T09:10:53.147 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 128, 2021-05-12T09:10:53.147 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 3 2021-05-12T09:10:53.147 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.148 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.148 INFO:tasks.workunit.client.0.smithi028.stdout: "poolid": 1, 2021-05-12T09:10:53.148 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 1, 2021-05-12T09:10:53.148 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:10:53.148 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 459280, 2021-05-12T09:10:53.149 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 2, 2021-05-12T09:10:53.149 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:10:53.149 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 6, 2021-05-12T09:10:53.149 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:10:53.149 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:10:53.150 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 2, 2021-05-12T09:10:53.150 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:10:53.150 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:10:53.150 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 2, 2021-05-12T09:10:53.150 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:10:53.151 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 76, 2021-05-12T09:10:53.151 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 64, 2021-05-12T09:10:53.151 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 95, 2021-05-12T09:10:53.151 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 1110, 2021-05-12T09:10:53.151 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:10:53.152 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:10:53.152 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:10:53.152 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:10:53.152 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:10:53.153 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:10:53.153 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:10:53.153 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:10:53.153 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:10:53.153 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:10:53.154 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:10:53.154 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:10:53.154 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:10:53.154 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:10:53.154 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:10:53.155 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:10:53.155 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:10:53.155 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:10:53.155 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:10:53.155 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:10:53.156 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:10:53.156 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:10:53.156 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:10:53.156 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:10:53.156 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:10:53.157 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.157 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:10:53.157 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:10:53.157 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:10:53.157 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:10:53.158 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 1388544, 2021-05-12T09:10:53.158 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1377840, 2021-05-12T09:10:53.158 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:10:53.158 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:10:53.159 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:10:53.159 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:10:53.159 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:10:53.159 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.160 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 14, 2021-05-12T09:10:53.160 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 14, 2021-05-12T09:10:53.160 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 2, 2021-05-12T09:10:53.160 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 2, 2021-05-12T09:10:53.161 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 3 2021-05-12T09:10:53.161 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.161 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:53.161 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_stats": [ 2021-05-12T09:10:53.161 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.162 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:10:53.162 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 184683593739 2021-05-12T09:10:53.162 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.162 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.162 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:10:53.163 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 193273528330 2021-05-12T09:10:53.163 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.163 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.163 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:10:53.163 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 64424509455 2021-05-12T09:10:53.164 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.164 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:53.164 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_by_state": [ 2021-05-12T09:10:53.164 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.164 INFO:tasks.workunit.client.0.smithi028.stdout: "state": "active+undersized+degraded", 2021-05-12T09:10:53.165 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 65 2021-05-12T09:10:53.165 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.165 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:53.165 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_by_osd": [ 2021-05-12T09:10:53.165 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.166 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:10:53.166 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:10:53.166 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 0, 2021-05-12T09:10:53.166 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:10:53.167 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.167 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.167 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:10:53.167 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 65, 2021-05-12T09:10:53.167 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 65, 2021-05-12T09:10:53.168 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:10:53.168 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.168 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.168 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:10:53.168 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:10:53.169 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 65, 2021-05-12T09:10:53.169 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:10:53.169 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.169 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:53.169 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": [ 2021-05-12T09:10:53.170 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.170 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 1, 2021-05-12T09:10:53.170 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": {} 2021-05-12T09:10:53.170 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.170 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:10:53.171 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 2, 2021-05-12T09:10:53.171 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": {} 2021-05-12T09:10:53.171 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.171 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:10:53.172 INFO:tasks.workunit.client.0.smithi028.stdout: "servicemap": { 2021-05-12T09:10:53.172 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 5, 2021-05-12T09:10:53.172 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:10:22.042900+0000", 2021-05-12T09:10:53.172 INFO:tasks.workunit.client.0.smithi028.stdout: "services": { 2021-05-12T09:10:53.172 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": { 2021-05-12T09:10:53.173 INFO:tasks.workunit.client.0.smithi028.stdout: "daemons": { 2021-05-12T09:10:53.173 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": "", 2021-05-12T09:10:53.173 INFO:tasks.workunit.client.0.smithi028.stdout: "0": { 2021-05-12T09:10:53.173 INFO:tasks.workunit.client.0.smithi028.stdout: "start_epoch": 0, 2021-05-12T09:10:53.174 INFO:tasks.workunit.client.0.smithi028.stdout: "start_stamp": "0.000000", 2021-05-12T09:10:53.174 INFO:tasks.workunit.client.0.smithi028.stdout: "gid": 0, 2021-05-12T09:10:53.174 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T09:10:53.174 INFO:tasks.workunit.client.0.smithi028.stdout: "metadata": {}, 2021-05-12T09:10:53.175 INFO:tasks.workunit.client.0.smithi028.stdout: "task_status": {} 2021-05-12T09:10:53.175 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.175 INFO:tasks.workunit.client.0.smithi028.stdout: "1": { 2021-05-12T09:10:53.175 INFO:tasks.workunit.client.0.smithi028.stdout: "start_epoch": 0, 2021-05-12T09:10:53.175 INFO:tasks.workunit.client.0.smithi028.stdout: "start_stamp": "0.000000", 2021-05-12T09:10:53.176 INFO:tasks.workunit.client.0.smithi028.stdout: "gid": 0, 2021-05-12T09:10:53.176 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "(unrecognized address family 0)/0", 2021-05-12T09:10:53.176 INFO:tasks.workunit.client.0.smithi028.stdout: "metadata": {}, 2021-05-12T09:10:53.176 INFO:tasks.workunit.client.0.smithi028.stdout: "task_status": {} 2021-05-12T09:10:53.176 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.177 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.177 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.177 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.177 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:10:53.178 INFO:tasks.workunit.client.0.smithi028.stdout: "paxos": { 2021-05-12T09:10:53.178 INFO:tasks.workunit.client.0.smithi028.stdout: "first_committed": 1, 2021-05-12T09:10:53.178 INFO:tasks.workunit.client.0.smithi028.stdout: "last_committed": 386, 2021-05-12T09:10:53.178 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pn": 0, 2021-05-12T09:10:53.178 INFO:tasks.workunit.client.0.smithi028.stdout: "accepted_pn": 0 2021-05-12T09:10:53.179 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:10:53.179 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:10:53.180 INFO:tasks.workunit.client.0.smithi028.stderr:report 3517580712 2021-05-12T09:10:53.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T09:10:53.181 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2021-05-12T09:10:53.181 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2021-05-12T09:10:53.181 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2021-05-12T09:10:53.181 INFO:tasks.workunit.client.0.smithi028.stdout:writing non-divergent object existing_4 2021-05-12T09:10:53.182 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2021-05-12T09:10:53.228 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:10:53.229 INFO:tasks.workunit.client.0.smithi028.stdout:2.3f 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325151+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.229 INFO:tasks.workunit.client.0.smithi028.stdout:2.3e 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.352066+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.229 INFO:tasks.workunit.client.0.smithi028.stdout:2.3d 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324201+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.229 INFO:tasks.workunit.client.0.smithi028.stdout:2.3c 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324071+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.230 INFO:tasks.workunit.client.0.smithi028.stdout:2.3b 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323529+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.230 INFO:tasks.workunit.client.0.smithi028.stdout:2.3a 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326590+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.230 INFO:tasks.workunit.client.0.smithi028.stdout:2.39 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318815+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.230 INFO:tasks.workunit.client.0.smithi028.stdout:2.38 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320936+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.231 INFO:tasks.workunit.client.0.smithi028.stdout:2.37 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321488+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.231 INFO:tasks.workunit.client.0.smithi028.stdout:2.36 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326290+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.231 INFO:tasks.workunit.client.0.smithi028.stdout:2.35 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319374+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.231 INFO:tasks.workunit.client.0.smithi028.stdout:2.34 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321682+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.231 INFO:tasks.workunit.client.0.smithi028.stdout:2.33 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326133+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.232 INFO:tasks.workunit.client.0.smithi028.stdout:2.32 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326137+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.232 INFO:tasks.workunit.client.0.smithi028.stdout:2.31 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319432+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.232 INFO:tasks.workunit.client.0.smithi028.stdout:2.30 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320564+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.232 INFO:tasks.workunit.client.0.smithi028.stdout:2.2f 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.327561+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.232 INFO:tasks.workunit.client.0.smithi028.stdout:2.2e 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325408+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.233 INFO:tasks.workunit.client.0.smithi028.stdout:2.2d 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318417+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.234 INFO:tasks.workunit.client.0.smithi028.stdout:2.2c 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320637+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.234 INFO:tasks.workunit.client.0.smithi028.stdout:2.2b 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326283+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.234 INFO:tasks.workunit.client.0.smithi028.stdout:2.2a 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.327146+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.234 INFO:tasks.workunit.client.0.smithi028.stdout:2.29 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320138+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.235 INFO:tasks.workunit.client.0.smithi028.stdout:2.28 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320304+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.235 INFO:tasks.workunit.client.0.smithi028.stdout:2.27 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325633+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.235 INFO:tasks.workunit.client.0.smithi028.stdout:2.26 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325591+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.236 INFO:tasks.workunit.client.0.smithi028.stdout:2.25 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326279+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.236 INFO:tasks.workunit.client.0.smithi028.stdout:2.24 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323195+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.236 INFO:tasks.workunit.client.0.smithi028.stdout:2.23 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325469+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.236 INFO:tasks.workunit.client.0.smithi028.stdout:2.e 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320245+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.236 INFO:tasks.workunit.client.0.smithi028.stdout:2.d 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321136+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.237 INFO:tasks.workunit.client.0.smithi028.stdout:2.c 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.322090+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.237 INFO:tasks.workunit.client.0.smithi028.stdout:2.b 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319604+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.237 INFO:tasks.workunit.client.0.smithi028.stdout:2.a 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319929+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.237 INFO:tasks.workunit.client.0.smithi028.stdout:2.9 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323963+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.238 INFO:tasks.workunit.client.0.smithi028.stdout:2.8 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323896+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.238 INFO:tasks.workunit.client.0.smithi028.stdout:2.7 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325446+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.238 INFO:tasks.workunit.client.0.smithi028.stdout:2.6 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.322282+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.239 INFO:tasks.workunit.client.0.smithi028.stdout:2.5 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318856+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.239 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321112+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.239 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:10:17.320695+0000 33'62 51:116 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:20.116394+0000 0'0 2021-05-12T09:09:20.116394+0000 0 2021-05-12T09:10:53.240 INFO:tasks.workunit.client.0.smithi028.stdout:2.3 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321973+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.240 INFO:tasks.workunit.client.0.smithi028.stdout:2.1 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321360+0000 33'100 51:123 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.240 INFO:tasks.workunit.client.0.smithi028.stdout:2.2 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323329+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.241 INFO:tasks.workunit.client.0.smithi028.stdout:2.4 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321691+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.241 INFO:tasks.workunit.client.0.smithi028.stdout:2.f 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.220557+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.241 INFO:tasks.workunit.client.0.smithi028.stdout:2.10 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321851+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.242 INFO:tasks.workunit.client.0.smithi028.stdout:2.11 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324153+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.242 INFO:tasks.workunit.client.0.smithi028.stdout:2.12 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320164+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.242 INFO:tasks.workunit.client.0.smithi028.stdout:2.13 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318763+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.242 INFO:tasks.workunit.client.0.smithi028.stdout:2.14 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319539+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.243 INFO:tasks.workunit.client.0.smithi028.stdout:2.15 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318618+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.243 INFO:tasks.workunit.client.0.smithi028.stdout:2.16 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321253+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.243 INFO:tasks.workunit.client.0.smithi028.stdout:2.17 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.220474+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.243 INFO:tasks.workunit.client.0.smithi028.stdout:2.18 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319857+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.243 INFO:tasks.workunit.client.0.smithi028.stdout:2.19 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321903+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.244 INFO:tasks.workunit.client.0.smithi028.stdout:2.1a 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320526+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.244 INFO:tasks.workunit.client.0.smithi028.stdout:2.1b 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318500+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.244 INFO:tasks.workunit.client.0.smithi028.stdout:2.1c 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319781+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.244 INFO:tasks.workunit.client.0.smithi028.stdout:2.1d 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318542+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.245 INFO:tasks.workunit.client.0.smithi028.stdout:2.1e 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323197+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.245 INFO:tasks.workunit.client.0.smithi028.stdout:2.1f 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323951+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.245 INFO:tasks.workunit.client.0.smithi028.stdout:2.20 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323744+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.245 INFO:tasks.workunit.client.0.smithi028.stdout:2.21 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323484+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.245 INFO:tasks.workunit.client.0.smithi028.stdout:2.22 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325216+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:10:53.246 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:10:53.246 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:10:53.247 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:10:53.247 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2021-05-12T09:10:53.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T09:10:53.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2021-05-12T09:10:53.358 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:10:53.358 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:10:53.358 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:10:53.358 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:10:53.359 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T09:10:53.360 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:10:53.360 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:10:53.361 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:10:53.361 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:10:53.361 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:10:53.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:10:53.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:10:53.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:10:53.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:10:53.417 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:10:53.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:10:53.801 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:10:53.802 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:10:53.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:10:53.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:10:53.802 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:10:53.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=184683593741 2021-05-12T09:10:53.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 184683593741 2021-05-12T09:10:53.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-184683593741' 2021-05-12T09:10:53.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:10:53.953 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:10:54.093 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=193273528332 2021-05-12T09:10:54.094 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 193273528332 2021-05-12T09:10:54.094 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-184683593741 1-193273528332' 2021-05-12T09:10:54.094 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:10:54.095 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:10:54.197 INFO:tasks.workunit.client.0.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-05-12T09:10:54.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T09:10:54.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T09:10:54.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T09:10:54.200 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:10:54.200 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-184683593741 2021-05-12T09:10:54.200 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:10:54.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:10:54.202 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-184683593741 2021-05-12T09:10:54.202 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:10:54.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=184683593741 2021-05-12T09:10:54.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 184683593741' 2021-05-12T09:10:54.203 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 184683593741 2021-05-12T09:10:54.204 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:10:54.587 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 184683593741 -lt 184683593741 2021-05-12T09:10:54.587 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:10:54.588 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-193273528332 2021-05-12T09:10:54.588 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:10:54.588 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:10:54.589 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-193273528332 2021-05-12T09:10:54.589 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:10:54.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=193273528332 2021-05-12T09:10:54.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 193273528332' 2021-05-12T09:10:54.590 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 193273528332 2021-05-12T09:10:54.591 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:10:54.976 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 193273528330 -lt 193273528332 2021-05-12T09:10:54.977 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:10:55.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:10:55.978 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:10:56.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 193273528333 -lt 193273528332 2021-05-12T09:10:56.382 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:10:56.382 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:10:56.382 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:10:56.891 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 65 == 0 2021-05-12T09:10:56.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:10:56.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:10:56.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:10:56.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:10:56.893 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:10:56.893 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:10:56.893 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:10:57.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:10:57.277 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:10:57.278 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:10:57.278 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:10:57.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:10:57.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T09:10:57.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T09:10:57.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T09:10:57.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:10:57.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:10:57.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:10:57.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:10:57.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:10:57.893 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:10:57.893 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:10:57.893 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:10:57.893 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:10:57.894 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:10:58.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:10:58.281 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:10:58.282 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:10:58.282 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:10:58.795 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:10:58.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:10:58.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:10:58.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:10:58.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:10:58.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:10:58.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:10:58.799 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:10:58.800 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:10:59.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:10:59.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:10:59.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T09:10:59.303 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:10:59.303 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T09:10:59.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:10:59.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:10:59.502 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:10:59.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:10:59.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:10:59.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:10:59.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:10:59.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:10:59.887 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:10:59.887 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:10:59.888 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:10:59.888 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:11:00.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:11:00.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:11:00.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:11:00.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:11:00.401 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:11:00.401 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:11:00.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:11:00.402 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:11:00.402 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:11:00.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:11:00.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:11:00.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T09:11:00.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:11:00.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T09:11:01.306 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:11:01.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:11:01.307 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:11:01.307 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:11:01.307 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:11:01.308 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:11:01.308 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:11:01.308 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:11:01.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:11:01.701 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:11:01.701 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:11:01.701 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:11:02.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:11:02.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:11:02.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:11:02.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:11:02.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:11:02.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:11:02.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:11:02.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:11:02.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:11:02.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:11:02.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:11:02.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T09:11:02.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:11:02.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T09:11:03.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:11:03.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:11:03.510 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:11:03.510 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:11:03.510 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:11:03.510 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:11:03.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:11:03.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:11:03.908 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:11:03.909 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:11:03.909 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:11:03.909 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:11:04.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:11:04.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:11:04.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:11:04.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:11:04.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:11:04.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:11:04.423 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:11:04.423 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:11:04.423 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:11:04.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:11:04.924 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:11:04.925 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T09:11:04.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:11:04.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T09:11:06.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:11:06.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:11:06.526 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:11:06.527 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:11:06.527 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:11:06.527 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:11:06.528 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:11:06.528 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:11:06.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:11:06.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:11:06.924 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:11:06.924 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:11:07.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:11:07.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:11:07.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:11:07.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:11:07.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:11:07.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:11:07.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:11:07.437 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:11:07.438 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:11:07.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:11:07.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:11:07.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T09:11:07.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:11:07.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T09:11:11.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:11:11.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:11:11.144 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:11:11.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:11:11.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:11:11.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:11:11.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:11:11.146 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:11:11.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:11:11.544 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:11:11.544 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:11:11.544 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:11:12.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:11:12.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:11:12.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:11:12.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:11:12.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:11:12.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:11:12.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:11:12.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:11:12.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:11:12.589 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:11:12.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:11:12.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T09:11:12.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:11:12.591 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T09:11:18.990 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:11:18.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:11:18.992 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:11:18.992 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:11:18.992 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:11:18.993 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:11:18.993 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:11:18.993 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:11:19.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:11:19.390 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:11:19.390 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:11:19.391 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:11:19.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:11:19.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:11:19.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:11:19.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:11:19.908 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:11:19.908 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:11:19.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:11:19.909 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:11:19.910 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:11:20.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:11:20.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:11:20.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T09:11:20.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:11:20.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T09:11:27.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:11:27.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:11:27.743 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:11:27.743 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:11:27.743 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:11:27.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:11:27.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:11:27.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:11:28.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:11:28.140 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:11:28.141 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:11:28.141 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:11:28.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 65 2021-05-12T09:11:28.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:11:28.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:11:28.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:11:28.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:11:28.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:11:28.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:11:28.653 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:11:28.653 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:11:29.159 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:11:29.159 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:11:29.159 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T09:11:29.160 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T09:11:29.654 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:11:29.655 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_fingerprint": "69070f6e-49a7-4aa1-b600-c76826add343", 2021-05-12T09:11:29.655 INFO:tasks.workunit.client.0.smithi028.stdout: "version": "17.0.0-4095-ga8c673d5", 2021-05-12T09:11:29.655 INFO:tasks.workunit.client.0.smithi028.stdout: "commit": "a8c673d555488beec9ccda5812ba822e69c12ea9", 2021-05-12T09:11:29.656 INFO:tasks.workunit.client.0.smithi028.stdout: "timestamp": "2021-05-12T09:11:29.653018+0000", 2021-05-12T09:11:29.656 INFO:tasks.workunit.client.0.smithi028.stdout: "tag": "", 2021-05-12T09:11:29.656 INFO:tasks.workunit.client.0.smithi028.stdout: "health": { 2021-05-12T09:11:29.656 INFO:tasks.workunit.client.0.smithi028.stdout: "status": "HEALTH_WARN", 2021-05-12T09:11:29.656 INFO:tasks.workunit.client.0.smithi028.stdout: "checks": { 2021-05-12T09:11:29.657 INFO:tasks.workunit.client.0.smithi028.stdout: "OSDMAP_FLAGS": { 2021-05-12T09:11:29.657 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:11:29.657 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:11:29.657 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T09:11:29.658 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 17 2021-05-12T09:11:29.658 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.658 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [], 2021-05-12T09:11:29.659 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:11:29.659 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.659 INFO:tasks.workunit.client.0.smithi028.stdout: "OSD_DOWN": { 2021-05-12T09:11:29.660 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:11:29.660 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:11:29.660 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "1 osds down", 2021-05-12T09:11:29.660 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 1 2021-05-12T09:11:29.661 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.661 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:11:29.661 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.662 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "osd.2 (root=default,host=smithi028) is down" 2021-05-12T09:11:29.662 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.662 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.662 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:11:29.663 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.663 INFO:tasks.workunit.client.0.smithi028.stdout: "PG_DEGRADED": { 2021-05-12T09:11:29.663 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:11:29.663 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:11:29.663 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "Degraded data redundancy: 102/306 objects degraded (33.333%), 65 pgs degraded, 65 pgs undersized", 2021-05-12T09:11:29.664 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 130 2021-05-12T09:11:29.664 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.664 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:11:29.664 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.665 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 1.0 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.665 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.665 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.665 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.0 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.666 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.666 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.666 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.1 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.666 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.667 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.667 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.2 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.667 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.667 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.667 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.3 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.668 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.668 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.668 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.4 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.669 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.669 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.669 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.5 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.669 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.670 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.670 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.6 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.670 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.670 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.671 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.7 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.671 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.671 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.671 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.8 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.672 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.672 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.673 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.9 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.673 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.673 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.673 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.a is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.674 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.674 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.674 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.b is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.674 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.675 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.675 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.c is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.675 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.676 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.676 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.d is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.676 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.676 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.677 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.e is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.677 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.677 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.677 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.f is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.678 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.678 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.678 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.10 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.678 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.679 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.679 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.11 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.679 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.679 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.679 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.12 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.680 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.680 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.680 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.13 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.680 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.681 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.681 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.14 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:11:29.681 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.681 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.682 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.23 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.682 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.683 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.683 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.24 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.683 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.683 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.684 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.25 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.684 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.684 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.685 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.26 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.685 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.685 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.686 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.27 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.686 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.686 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.686 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.28 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.687 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.687 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.687 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.29 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.687 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.688 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.688 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.2a is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.688 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.689 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.689 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.2b is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.689 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.689 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.690 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.2c is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.690 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.690 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.690 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.2d is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.691 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.691 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.691 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.2e is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.691 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.691 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.692 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.2f is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.692 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.692 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.692 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.30 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.693 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.693 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.693 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.31 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.694 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.694 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.694 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.32 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.694 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.695 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.695 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.33 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.695 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.695 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.696 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.34 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.696 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.696 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.696 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.35 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.697 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.697 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.697 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.36 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.697 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.698 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.698 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.37 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.698 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.698 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.699 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.38 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.699 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.699 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.699 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.39 is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.700 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.700 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.700 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.3a is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.700 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.701 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.701 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.3b is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.701 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.701 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.701 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.3c is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.702 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.702 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.702 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.3d is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.702 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.703 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.703 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.3e is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.703 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.703 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.704 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.3f is stuck undersized for 71s, current state active+undersized+degraded, last acting [1,0]" 2021-05-12T09:11:29.704 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.704 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.704 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:11:29.704 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.705 INFO:tasks.workunit.client.0.smithi028.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T09:11:29.705 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:11:29.705 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:11:29.705 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T09:11:29.706 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 1 2021-05-12T09:11:29.706 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.706 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:11:29.707 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.707 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T09:11:29.707 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.707 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.708 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T09:11:29.708 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.708 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.708 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:11:29.709 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.709 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.709 INFO:tasks.workunit.client.0.smithi028.stdout: "mutes": [] 2021-05-12T09:11:29.709 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.710 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap_first_committed": 1, 2021-05-12T09:11:29.710 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap_last_committed": 1, 2021-05-12T09:11:29.710 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap": { 2021-05-12T09:11:29.710 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 1, 2021-05-12T09:11:29.711 INFO:tasks.workunit.client.0.smithi028.stdout: "fsid": "8342ec9e-531f-44b4-8c8c-079d667fe908", 2021-05-12T09:11:29.711 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:08:51.540323Z", 2021-05-12T09:11:29.711 INFO:tasks.workunit.client.0.smithi028.stdout: "created": "2021-05-12T09:08:51.540323Z", 2021-05-12T09:11:29.711 INFO:tasks.workunit.client.0.smithi028.stdout: "min_mon_release": 17, 2021-05-12T09:11:29.712 INFO:tasks.workunit.client.0.smithi028.stdout: "min_mon_release_name": "quincy", 2021-05-12T09:11:29.712 INFO:tasks.workunit.client.0.smithi028.stdout: "election_strategy": 1, 2021-05-12T09:11:29.712 INFO:tasks.workunit.client.0.smithi028.stdout: "disallowed_leaders: ": "", 2021-05-12T09:11:29.712 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode": false, 2021-05-12T09:11:29.713 INFO:tasks.workunit.client.0.smithi028.stdout: "features": { 2021-05-12T09:11:29.713 INFO:tasks.workunit.client.0.smithi028.stdout: "persistent": [ 2021-05-12T09:11:29.713 INFO:tasks.workunit.client.0.smithi028.stdout: "kraken", 2021-05-12T09:11:29.713 INFO:tasks.workunit.client.0.smithi028.stdout: "luminous", 2021-05-12T09:11:29.714 INFO:tasks.workunit.client.0.smithi028.stdout: "mimic", 2021-05-12T09:11:29.714 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap-prune", 2021-05-12T09:11:29.714 INFO:tasks.workunit.client.0.smithi028.stdout: "nautilus", 2021-05-12T09:11:29.714 INFO:tasks.workunit.client.0.smithi028.stdout: "octopus", 2021-05-12T09:11:29.715 INFO:tasks.workunit.client.0.smithi028.stdout: "pacific", 2021-05-12T09:11:29.715 INFO:tasks.workunit.client.0.smithi028.stdout: "elector-pinging", 2021-05-12T09:11:29.715 INFO:tasks.workunit.client.0.smithi028.stdout: "quincy" 2021-05-12T09:11:29.715 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.716 INFO:tasks.workunit.client.0.smithi028.stdout: "optional": [] 2021-05-12T09:11:29.716 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.716 INFO:tasks.workunit.client.0.smithi028.stdout: "mons": [ 2021-05-12T09:11:29.716 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.716 INFO:tasks.workunit.client.0.smithi028.stdout: "rank": 0, 2021-05-12T09:11:29.717 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "a", 2021-05-12T09:11:29.717 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:11:29.717 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:11:29.718 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.718 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:11:29.718 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:7115", 2021-05-12T09:11:29.719 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 0 2021-05-12T09:11:29.719 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.719 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.719 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.720 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T09:11:29.720 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T09:11:29.720 INFO:tasks.workunit.client.0.smithi028.stdout: "priority": 0, 2021-05-12T09:11:29.720 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 0, 2021-05-12T09:11:29.720 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_location": "{}" 2021-05-12T09:11:29.721 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.721 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.721 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.721 INFO:tasks.workunit.client.0.smithi028.stdout: "quorum": [ 2021-05-12T09:11:29.722 INFO:tasks.workunit.client.0.smithi028.stdout: 0 2021-05-12T09:11:29.722 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.722 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap": { 2021-05-12T09:11:29.722 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 51, 2021-05-12T09:11:29.723 INFO:tasks.workunit.client.0.smithi028.stdout: "fsid": "8342ec9e-531f-44b4-8c8c-079d667fe908", 2021-05-12T09:11:29.723 INFO:tasks.workunit.client.0.smithi028.stdout: "created": "2021-05-12T09:08:52.384986+0000", 2021-05-12T09:11:29.723 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:10:16.883217+0000", 2021-05-12T09:11:29.723 INFO:tasks.workunit.client.0.smithi028.stdout: "last_up_change": "2021-05-12T09:10:14.790190+0000", 2021-05-12T09:11:29.723 INFO:tasks.workunit.client.0.smithi028.stdout: "last_in_change": "2021-05-12T09:10:16.072476+0000", 2021-05-12T09:11:29.724 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T09:11:29.724 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_num": 5800384, 2021-05-12T09:11:29.724 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_set": [ 2021-05-12T09:11:29.724 INFO:tasks.workunit.client.0.smithi028.stdout: "nodown", 2021-05-12T09:11:29.725 INFO:tasks.workunit.client.0.smithi028.stdout: "noin", 2021-05-12T09:11:29.725 INFO:tasks.workunit.client.0.smithi028.stdout: "noout", 2021-05-12T09:11:29.725 INFO:tasks.workunit.client.0.smithi028.stdout: "pglog_hardlimit", 2021-05-12T09:11:29.725 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snapdirs", 2021-05-12T09:11:29.725 INFO:tasks.workunit.client.0.smithi028.stdout: "recovery_deletes", 2021-05-12T09:11:29.726 INFO:tasks.workunit.client.0.smithi028.stdout: "sortbitwise" 2021-05-12T09:11:29.726 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.726 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_version": 9, 2021-05-12T09:11:29.726 INFO:tasks.workunit.client.0.smithi028.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T09:11:29.727 INFO:tasks.workunit.client.0.smithi028.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T09:11:29.727 INFO:tasks.workunit.client.0.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T09:11:29.727 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_snapshot": "", 2021-05-12T09:11:29.727 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_max": 2, 2021-05-12T09:11:29.728 INFO:tasks.workunit.client.0.smithi028.stdout: "max_osd": 3, 2021-05-12T09:11:29.728 INFO:tasks.workunit.client.0.smithi028.stdout: "require_min_compat_client": "luminous", 2021-05-12T09:11:29.728 INFO:tasks.workunit.client.0.smithi028.stdout: "min_compat_client": "jewel", 2021-05-12T09:11:29.728 INFO:tasks.workunit.client.0.smithi028.stdout: "require_osd_release": "quincy", 2021-05-12T09:11:29.729 INFO:tasks.workunit.client.0.smithi028.stdout: "pools": [ 2021-05-12T09:11:29.729 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.729 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 1, 2021-05-12T09:11:29.729 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_name": ".mgr", 2021-05-12T09:11:29.730 INFO:tasks.workunit.client.0.smithi028.stdout: "create_time": "2021-05-12T09:09:20.066263+0000", 2021-05-12T09:11:29.730 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": 1, 2021-05-12T09:11:29.730 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_names": "hashpspool", 2021-05-12T09:11:29.730 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:11:29.731 INFO:tasks.workunit.client.0.smithi028.stdout: "size": 3, 2021-05-12T09:11:29.731 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 2, 2021-05-12T09:11:29.731 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_rule": 0, 2021-05-12T09:11:29.731 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_count": 0, 2021-05-12T09:11:29.732 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_target": 0, 2021-05-12T09:11:29.732 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T09:11:29.732 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T09:11:29.732 INFO:tasks.workunit.client.0.smithi028.stdout: "object_hash": 2, 2021-05-12T09:11:29.733 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_autoscale_mode": "off", 2021-05-12T09:11:29.733 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num": 1, 2021-05-12T09:11:29.733 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num": 1, 2021-05-12T09:11:29.733 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num_target": 1, 2021-05-12T09:11:29.734 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_target": 1, 2021-05-12T09:11:29.734 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_pending": 1, 2021-05-12T09:11:29.734 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pg_merge_meta": { 2021-05-12T09:11:29.734 INFO:tasks.workunit.client.0.smithi028.stdout: "source_pgid": "0.0", 2021-05-12T09:11:29.735 INFO:tasks.workunit.client.0.smithi028.stdout: "ready_epoch": 0, 2021-05-12T09:11:29.735 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_started": 0, 2021-05-12T09:11:29.735 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": 0, 2021-05-12T09:11:29.735 INFO:tasks.workunit.client.0.smithi028.stdout: "source_version": "0'0", 2021-05-12T09:11:29.736 INFO:tasks.workunit.client.0.smithi028.stdout: "target_version": "0'0" 2021-05-12T09:11:29.736 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.736 INFO:tasks.workunit.client.0.smithi028.stdout: "last_change": "20", 2021-05-12T09:11:29.736 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend": "0", 2021-05-12T09:11:29.737 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T09:11:29.737 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T09:11:29.737 INFO:tasks.workunit.client.0.smithi028.stdout: "auid": 0, 2021-05-12T09:11:29.738 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_mode": "selfmanaged", 2021-05-12T09:11:29.738 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_seq": 0, 2021-05-12T09:11:29.738 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_epoch": 0, 2021-05-12T09:11:29.738 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_snaps": [], 2021-05-12T09:11:29.739 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps": "[]", 2021-05-12T09:11:29.739 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_bytes": 0, 2021-05-12T09:11:29.739 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_objects": 0, 2021-05-12T09:11:29.739 INFO:tasks.workunit.client.0.smithi028.stdout: "tiers": [], 2021-05-12T09:11:29.740 INFO:tasks.workunit.client.0.smithi028.stdout: "tier_of": -1, 2021-05-12T09:11:29.740 INFO:tasks.workunit.client.0.smithi028.stdout: "read_tier": -1, 2021-05-12T09:11:29.740 INFO:tasks.workunit.client.0.smithi028.stdout: "write_tier": -1, 2021-05-12T09:11:29.740 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_mode": "none", 2021-05-12T09:11:29.741 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_bytes": 0, 2021-05-12T09:11:29.741 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_objects": 0, 2021-05-12T09:11:29.741 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T09:11:29.741 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T09:11:29.741 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T09:11:29.742 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_flush_age": 0, 2021-05-12T09:11:29.742 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_evict_age": 0, 2021-05-12T09:11:29.742 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profile": "", 2021-05-12T09:11:29.742 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_params": { 2021-05-12T09:11:29.743 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "none" 2021-05-12T09:11:29.743 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.743 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_period": 0, 2021-05-12T09:11:29.744 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_count": 0, 2021-05-12T09:11:29.744 INFO:tasks.workunit.client.0.smithi028.stdout: "use_gmt_hitset": true, 2021-05-12T09:11:29.744 INFO:tasks.workunit.client.0.smithi028.stdout: "min_read_recency_for_promote": 0, 2021-05-12T09:11:29.744 INFO:tasks.workunit.client.0.smithi028.stdout: "min_write_recency_for_promote": 0, 2021-05-12T09:11:29.745 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T09:11:29.745 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_search_last_n": 0, 2021-05-12T09:11:29.745 INFO:tasks.workunit.client.0.smithi028.stdout: "grade_table": [], 2021-05-12T09:11:29.745 INFO:tasks.workunit.client.0.smithi028.stdout: "stripe_width": 0, 2021-05-12T09:11:29.746 INFO:tasks.workunit.client.0.smithi028.stdout: "expected_num_objects": 0, 2021-05-12T09:11:29.746 INFO:tasks.workunit.client.0.smithi028.stdout: "fast_read": false, 2021-05-12T09:11:29.746 INFO:tasks.workunit.client.0.smithi028.stdout: "options": { 2021-05-12T09:11:29.746 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_min": 1 2021-05-12T09:11:29.747 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.747 INFO:tasks.workunit.client.0.smithi028.stdout: "application_metadata": { 2021-05-12T09:11:29.747 INFO:tasks.workunit.client.0.smithi028.stdout: "mgr": {} 2021-05-12T09:11:29.747 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.747 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.748 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.748 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 2, 2021-05-12T09:11:29.748 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_name": "test", 2021-05-12T09:11:29.749 INFO:tasks.workunit.client.0.smithi028.stdout: "create_time": "2021-05-12T09:09:23.059340+0000", 2021-05-12T09:11:29.749 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": 1, 2021-05-12T09:11:29.749 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_names": "hashpspool", 2021-05-12T09:11:29.749 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:11:29.750 INFO:tasks.workunit.client.0.smithi028.stdout: "size": 3, 2021-05-12T09:11:29.750 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 2, 2021-05-12T09:11:29.750 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_rule": 0, 2021-05-12T09:11:29.751 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_count": 0, 2021-05-12T09:11:29.751 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_target": 0, 2021-05-12T09:11:29.751 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T09:11:29.751 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T09:11:29.752 INFO:tasks.workunit.client.0.smithi028.stdout: "object_hash": 2, 2021-05-12T09:11:29.752 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_autoscale_mode": "on", 2021-05-12T09:11:29.752 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num": 64, 2021-05-12T09:11:29.752 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num": 1, 2021-05-12T09:11:29.753 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num_target": 64, 2021-05-12T09:11:29.753 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_target": 64, 2021-05-12T09:11:29.753 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_pending": 64, 2021-05-12T09:11:29.753 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pg_merge_meta": { 2021-05-12T09:11:29.754 INFO:tasks.workunit.client.0.smithi028.stdout: "source_pgid": "0.0", 2021-05-12T09:11:29.754 INFO:tasks.workunit.client.0.smithi028.stdout: "ready_epoch": 0, 2021-05-12T09:11:29.754 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_started": 0, 2021-05-12T09:11:29.754 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": 0, 2021-05-12T09:11:29.754 INFO:tasks.workunit.client.0.smithi028.stdout: "source_version": "0'0", 2021-05-12T09:11:29.755 INFO:tasks.workunit.client.0.smithi028.stdout: "target_version": "0'0" 2021-05-12T09:11:29.755 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.755 INFO:tasks.workunit.client.0.smithi028.stdout: "last_change": "35", 2021-05-12T09:11:29.755 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend": "0", 2021-05-12T09:11:29.756 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T09:11:29.756 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_preluminous": "35", 2021-05-12T09:11:29.756 INFO:tasks.workunit.client.0.smithi028.stdout: "auid": 0, 2021-05-12T09:11:29.756 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_mode": "selfmanaged", 2021-05-12T09:11:29.757 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_seq": 0, 2021-05-12T09:11:29.757 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_epoch": 0, 2021-05-12T09:11:29.757 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_snaps": [], 2021-05-12T09:11:29.757 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps": "[]", 2021-05-12T09:11:29.758 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_bytes": 0, 2021-05-12T09:11:29.758 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_objects": 0, 2021-05-12T09:11:29.758 INFO:tasks.workunit.client.0.smithi028.stdout: "tiers": [], 2021-05-12T09:11:29.759 INFO:tasks.workunit.client.0.smithi028.stdout: "tier_of": -1, 2021-05-12T09:11:29.759 INFO:tasks.workunit.client.0.smithi028.stdout: "read_tier": -1, 2021-05-12T09:11:29.759 INFO:tasks.workunit.client.0.smithi028.stdout: "write_tier": -1, 2021-05-12T09:11:29.759 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_mode": "none", 2021-05-12T09:11:29.760 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_bytes": 0, 2021-05-12T09:11:29.760 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_objects": 0, 2021-05-12T09:11:29.760 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T09:11:29.760 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T09:11:29.761 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T09:11:29.761 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_flush_age": 0, 2021-05-12T09:11:29.761 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_evict_age": 0, 2021-05-12T09:11:29.761 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profile": "", 2021-05-12T09:11:29.762 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_params": { 2021-05-12T09:11:29.762 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "none" 2021-05-12T09:11:29.762 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.762 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_period": 0, 2021-05-12T09:11:29.763 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_count": 0, 2021-05-12T09:11:29.763 INFO:tasks.workunit.client.0.smithi028.stdout: "use_gmt_hitset": true, 2021-05-12T09:11:29.763 INFO:tasks.workunit.client.0.smithi028.stdout: "min_read_recency_for_promote": 0, 2021-05-12T09:11:29.763 INFO:tasks.workunit.client.0.smithi028.stdout: "min_write_recency_for_promote": 0, 2021-05-12T09:11:29.763 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T09:11:29.764 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_search_last_n": 0, 2021-05-12T09:11:29.764 INFO:tasks.workunit.client.0.smithi028.stdout: "grade_table": [], 2021-05-12T09:11:29.764 INFO:tasks.workunit.client.0.smithi028.stdout: "stripe_width": 0, 2021-05-12T09:11:29.764 INFO:tasks.workunit.client.0.smithi028.stdout: "expected_num_objects": 0, 2021-05-12T09:11:29.765 INFO:tasks.workunit.client.0.smithi028.stdout: "fast_read": false, 2021-05-12T09:11:29.765 INFO:tasks.workunit.client.0.smithi028.stdout: "options": {}, 2021-05-12T09:11:29.765 INFO:tasks.workunit.client.0.smithi028.stdout: "application_metadata": {} 2021-05-12T09:11:29.765 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.765 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.766 INFO:tasks.workunit.client.0.smithi028.stdout: "osds": [ 2021-05-12T09:11:29.766 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.766 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:11:29.766 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "18cc91da-f8e6-4da2-beb3-4002e77c4ec1", 2021-05-12T09:11:29.767 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 1, 2021-05-12T09:11:29.767 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:11:29.767 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:11:29.767 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:11:29.768 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 5, 2021-05-12T09:11:29.768 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 33, 2021-05-12T09:11:29.768 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 43, 2021-05-12T09:11:29.768 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 48, 2021-05-12T09:11:29.769 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 37, 2021-05-12T09:11:29.769 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:11:29.769 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:11:29.769 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:11:29.769 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.770 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:11:29.770 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6802", 2021-05-12T09:11:29.770 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:11:29.770 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.771 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.771 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:11:29.771 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6803", 2021-05-12T09:11:29.771 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:11:29.772 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.772 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.772 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.772 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:11:29.772 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:11:29.773 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.773 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:11:29.773 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6804", 2021-05-12T09:11:29.773 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:11:29.774 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.774 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.774 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:11:29.774 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6805", 2021-05-12T09:11:29.775 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:11:29.775 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.775 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.775 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.776 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:11:29.776 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:11:29.776 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.777 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:11:29.777 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6808", 2021-05-12T09:11:29.777 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:11:29.777 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.778 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.778 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:11:29.778 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6809", 2021-05-12T09:11:29.778 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:11:29.778 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.779 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.779 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.779 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:11:29.779 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:11:29.780 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.780 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:11:29.780 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6806", 2021-05-12T09:11:29.780 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:11:29.781 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.781 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.781 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:11:29.781 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6807", 2021-05-12T09:11:29.781 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63042 2021-05-12T09:11:29.782 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.782 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.782 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.783 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6803/63042", 2021-05-12T09:11:29.783 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6805/63042", 2021-05-12T09:11:29.783 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6809/63042", 2021-05-12T09:11:29.783 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6807/63042", 2021-05-12T09:11:29.783 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:11:29.784 INFO:tasks.workunit.client.0.smithi028.stdout: "exists", 2021-05-12T09:11:29.784 INFO:tasks.workunit.client.0.smithi028.stdout: "up" 2021-05-12T09:11:29.784 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.784 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.785 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.785 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:11:29.785 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "602d76d3-2336-45c7-b45a-9e2846468b99", 2021-05-12T09:11:29.786 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 1, 2021-05-12T09:11:29.786 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:11:29.786 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:11:29.787 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:11:29.787 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 10, 2021-05-12T09:11:29.787 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 33, 2021-05-12T09:11:29.787 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 45, 2021-05-12T09:11:29.787 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 50, 2021-05-12T09:11:29.788 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 38, 2021-05-12T09:11:29.788 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:11:29.788 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:11:29.788 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:11:29.789 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.789 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:11:29.789 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6810", 2021-05-12T09:11:29.789 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:11:29.790 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.790 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.790 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:11:29.790 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6811", 2021-05-12T09:11:29.791 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:11:29.791 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.791 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.791 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.792 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:11:29.792 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:11:29.792 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.792 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:11:29.792 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6812", 2021-05-12T09:11:29.793 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:11:29.793 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.793 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.793 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:11:29.794 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6813", 2021-05-12T09:11:29.794 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:11:29.794 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.794 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.795 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.795 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:11:29.795 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:11:29.795 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.796 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:11:29.796 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6816", 2021-05-12T09:11:29.796 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:11:29.796 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.797 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.797 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:11:29.797 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6817", 2021-05-12T09:11:29.797 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:11:29.798 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.798 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.798 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.798 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:11:29.799 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:11:29.799 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.799 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:11:29.799 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6814", 2021-05-12T09:11:29.799 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:11:29.800 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.800 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.800 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:11:29.800 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6815", 2021-05-12T09:11:29.801 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 63599 2021-05-12T09:11:29.801 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.801 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.801 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.802 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6811/63599", 2021-05-12T09:11:29.802 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6813/63599", 2021-05-12T09:11:29.802 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6817/63599", 2021-05-12T09:11:29.802 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6815/63599", 2021-05-12T09:11:29.802 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:11:29.803 INFO:tasks.workunit.client.0.smithi028.stdout: "exists", 2021-05-12T09:11:29.803 INFO:tasks.workunit.client.0.smithi028.stdout: "up" 2021-05-12T09:11:29.803 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.803 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.804 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.804 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:11:29.804 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "cabac835-d2f0-45a7-ba67-30ece6d00d83", 2021-05-12T09:11:29.804 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 0, 2021-05-12T09:11:29.805 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:11:29.805 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:11:29.805 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:11:29.805 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 0, 2021-05-12T09:11:29.805 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 0, 2021-05-12T09:11:29.806 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 15, 2021-05-12T09:11:29.806 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 35, 2021-05-12T09:11:29.806 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 39, 2021-05-12T09:11:29.806 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:11:29.807 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:11:29.807 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:11:29.807 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.807 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:11:29.808 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6818", 2021-05-12T09:11:29.808 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:11:29.808 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.808 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.809 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:11:29.809 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6819", 2021-05-12T09:11:29.809 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:11:29.809 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.809 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.810 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.810 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:11:29.810 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:11:29.810 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.811 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:11:29.811 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6820", 2021-05-12T09:11:29.811 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:11:29.811 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.811 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.812 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:11:29.812 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6821", 2021-05-12T09:11:29.812 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:11:29.812 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.813 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.813 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.813 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:11:29.813 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:11:29.813 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.814 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:11:29.814 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6824", 2021-05-12T09:11:29.814 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:11:29.814 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.815 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.815 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:11:29.815 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6825", 2021-05-12T09:11:29.815 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:11:29.816 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.816 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.816 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.816 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:11:29.816 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:11:29.817 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.817 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:11:29.817 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6822", 2021-05-12T09:11:29.817 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:11:29.818 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.818 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.818 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:11:29.818 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6823", 2021-05-12T09:11:29.818 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 58916 2021-05-12T09:11:29.819 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.819 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.819 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.819 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6819/58916", 2021-05-12T09:11:29.820 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6821/58916", 2021-05-12T09:11:29.820 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6825/58916", 2021-05-12T09:11:29.820 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6823/58916", 2021-05-12T09:11:29.820 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:11:29.821 INFO:tasks.workunit.client.0.smithi028.stdout: "exists" 2021-05-12T09:11:29.821 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.821 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.821 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.821 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_xinfo": [ 2021-05-12T09:11:29.822 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.822 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:11:29.822 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:10:04.942621+0000", 2021-05-12T09:11:29.822 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:11:29.823 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:11:29.823 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:11:29.823 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:11:29.823 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:09:01.894764+0000", 2021-05-12T09:11:29.823 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:11:29.824 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.824 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.824 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:11:29.824 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:10:05.572036+0000", 2021-05-12T09:11:29.825 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:11:29.825 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:11:29.825 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:11:29.825 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:11:29.826 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:09:10.922242+0000", 2021-05-12T09:11:29.826 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:11:29.826 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.826 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.826 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:11:29.827 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:10:06.119786+0000", 2021-05-12T09:11:29.827 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:11:29.827 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:11:29.827 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:11:29.828 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:11:29.828 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:09:20.187472+0000", 2021-05-12T09:11:29.828 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:11:29.828 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.829 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.829 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_upmap": [], 2021-05-12T09:11:29.829 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_upmap_items": [], 2021-05-12T09:11:29.829 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_temp": [], 2021-05-12T09:11:29.830 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_temp": [], 2021-05-12T09:11:29.830 INFO:tasks.workunit.client.0.smithi028.stdout: "blocklist": {}, 2021-05-12T09:11:29.830 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profiles": { 2021-05-12T09:11:29.830 INFO:tasks.workunit.client.0.smithi028.stdout: "default": { 2021-05-12T09:11:29.831 INFO:tasks.workunit.client.0.smithi028.stdout: "k": "2", 2021-05-12T09:11:29.831 INFO:tasks.workunit.client.0.smithi028.stdout: "m": "2", 2021-05-12T09:11:29.831 INFO:tasks.workunit.client.0.smithi028.stdout: "plugin": "jerasure", 2021-05-12T09:11:29.831 INFO:tasks.workunit.client.0.smithi028.stdout: "technique": "reed_sol_van" 2021-05-12T09:11:29.831 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.832 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.832 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps_queue": [], 2021-05-12T09:11:29.832 INFO:tasks.workunit.client.0.smithi028.stdout: "new_removed_snaps": [], 2021-05-12T09:11:29.832 INFO:tasks.workunit.client.0.smithi028.stdout: "new_purged_snaps": [], 2021-05-12T09:11:29.833 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_node_flags": {}, 2021-05-12T09:11:29.833 INFO:tasks.workunit.client.0.smithi028.stdout: "device_class_flags": {}, 2021-05-12T09:11:29.833 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode": { 2021-05-12T09:11:29.833 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode_enabled": false, 2021-05-12T09:11:29.834 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_bucket_count": 0, 2021-05-12T09:11:29.834 INFO:tasks.workunit.client.0.smithi028.stdout: "degraded_stretch_mode": 0, 2021-05-12T09:11:29.834 INFO:tasks.workunit.client.0.smithi028.stdout: "recovering_stretch_mode": 0, 2021-05-12T09:11:29.834 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode_bucket": 0 2021-05-12T09:11:29.835 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.835 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.835 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_metadata": [ 2021-05-12T09:11:29.835 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.835 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:11:29.836 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:11:29.836 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6804/63042,v1:127.0.0.1:6805/63042]", 2021-05-12T09:11:29.836 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:11:29.836 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:11:29.837 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:11:29.837 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:11:29.837 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:11:29.837 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:11:29.837 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:11:29.838 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:11:29.838 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:11:29.838 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T09:11:29.838 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:11:29.839 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:11:29.839 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:11:29.839 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:11:29.839 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:11:29.840 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:11:29.840 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:11:29.840 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:11:29.840 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:11:29.840 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:11:29.841 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:11:29.841 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:11:29.841 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:11:29.841 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:11:29.842 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:11:29.842 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6802/63042,v1:127.0.0.1:6803/63042]", 2021-05-12T09:11:29.842 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:11:29.842 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/63042,v1:127.0.0.1:6809/63042]", 2021-05-12T09:11:29.843 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/63042,v1:127.0.0.1:6807/63042]", 2021-05-12T09:11:29.843 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:11:29.843 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:11:29.843 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:11:29.844 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:11:29.844 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:11:29.844 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:11:29.844 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:11:29.844 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:11:29.845 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:11:29.845 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T09:11:29.845 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:11:29.845 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:11:29.846 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:11:29.846 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.846 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.846 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:11:29.846 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:11:29.847 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6812/63599,v1:127.0.0.1:6813/63599]", 2021-05-12T09:11:29.847 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:11:29.847 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:11:29.847 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:11:29.848 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:11:29.848 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:11:29.848 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:11:29.848 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:11:29.849 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:11:29.849 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:11:29.849 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T09:11:29.849 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:11:29.849 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:11:29.850 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:11:29.850 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:11:29.850 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:11:29.850 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:11:29.851 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:11:29.851 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:11:29.851 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:11:29.851 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:11:29.852 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:11:29.852 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:11:29.852 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:11:29.852 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:11:29.853 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:11:29.853 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6810/63599,v1:127.0.0.1:6811/63599]", 2021-05-12T09:11:29.853 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:11:29.853 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/63599,v1:127.0.0.1:6817/63599]", 2021-05-12T09:11:29.853 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/63599,v1:127.0.0.1:6815/63599]", 2021-05-12T09:11:29.854 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:11:29.854 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:11:29.854 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:11:29.854 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:11:29.855 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:11:29.855 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:11:29.855 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:11:29.855 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:11:29.855 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:11:29.856 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T09:11:29.856 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:11:29.856 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:11:29.856 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:11:29.857 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.857 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.857 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:11:29.857 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:11:29.857 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6820/58916,v1:127.0.0.1:6821/58916]", 2021-05-12T09:11:29.858 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:11:29.858 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:11:29.858 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:11:29.858 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:11:29.859 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:11:29.859 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:11:29.859 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:11:29.859 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:11:29.860 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:11:29.860 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T09:11:29.860 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:11:29.860 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:11:29.861 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:11:29.861 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:11:29.861 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:11:29.861 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:11:29.861 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:11:29.862 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:11:29.862 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:11:29.862 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:11:29.862 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:11:29.863 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:11:29.863 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:11:29.863 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:11:29.863 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:11:29.864 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6818/58916,v1:127.0.0.1:6819/58916]", 2021-05-12T09:11:29.864 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:11:29.864 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/58916,v1:127.0.0.1:6825/58916]", 2021-05-12T09:11:29.864 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/58916,v1:127.0.0.1:6823/58916]", 2021-05-12T09:11:29.864 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:11:29.865 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:11:29.865 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:11:29.865 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:11:29.865 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:11:29.866 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:11:29.866 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:11:29.866 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:11:29.866 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:11:29.867 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T09:11:29.867 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:11:29.867 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:11:29.867 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:11:29.868 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.868 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.868 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_clean_epochs": { 2021-05-12T09:11:29.868 INFO:tasks.workunit.client.0.smithi028.stdout: "min_last_epoch_clean": 0, 2021-05-12T09:11:29.868 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": { 2021-05-12T09:11:29.869 INFO:tasks.workunit.client.0.smithi028.stdout: "per_pool": [] 2021-05-12T09:11:29.869 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.869 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_epochs": [ 2021-05-12T09:11:29.869 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.870 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:11:29.870 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 43 2021-05-12T09:11:29.870 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.870 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.870 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:11:29.871 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 46 2021-05-12T09:11:29.871 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.871 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.871 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:11:29.872 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 15 2021-05-12T09:11:29.872 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.872 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.872 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.873 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_first_committed": 1, 2021-05-12T09:11:29.873 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_last_committed": 51, 2021-05-12T09:11:29.873 INFO:tasks.workunit.client.0.smithi028.stdout: "crushmap": { 2021-05-12T09:11:29.873 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": [ 2021-05-12T09:11:29.874 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.874 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:11:29.874 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.0", 2021-05-12T09:11:29.874 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:11:29.874 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.875 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.875 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:11:29.875 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.1", 2021-05-12T09:11:29.875 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:11:29.876 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.876 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.876 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:11:29.876 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.2", 2021-05-12T09:11:29.877 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:11:29.877 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.877 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.877 INFO:tasks.workunit.client.0.smithi028.stdout: "types": [ 2021-05-12T09:11:29.878 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.878 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 0, 2021-05-12T09:11:29.878 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd" 2021-05-12T09:11:29.879 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.879 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.879 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:11:29.879 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "host" 2021-05-12T09:11:29.880 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.880 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.880 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 2, 2021-05-12T09:11:29.880 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "chassis" 2021-05-12T09:11:29.880 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.881 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.881 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 3, 2021-05-12T09:11:29.881 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "rack" 2021-05-12T09:11:29.881 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.882 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.882 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 4, 2021-05-12T09:11:29.882 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "row" 2021-05-12T09:11:29.882 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.883 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.883 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 5, 2021-05-12T09:11:29.883 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "pdu" 2021-05-12T09:11:29.883 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.884 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.884 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 6, 2021-05-12T09:11:29.884 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "pod" 2021-05-12T09:11:29.884 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.885 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.885 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 7, 2021-05-12T09:11:29.885 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "room" 2021-05-12T09:11:29.885 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.886 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.886 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 8, 2021-05-12T09:11:29.886 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "datacenter" 2021-05-12T09:11:29.886 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.887 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.887 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 9, 2021-05-12T09:11:29.887 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "zone" 2021-05-12T09:11:29.887 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.887 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.888 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 10, 2021-05-12T09:11:29.888 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "region" 2021-05-12T09:11:29.888 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.888 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.889 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:11:29.889 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "root" 2021-05-12T09:11:29.889 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.889 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.890 INFO:tasks.workunit.client.0.smithi028.stdout: "buckets": [ 2021-05-12T09:11:29.890 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.890 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -1, 2021-05-12T09:11:29.890 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "default", 2021-05-12T09:11:29.890 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:11:29.891 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "root", 2021-05-12T09:11:29.891 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:11:29.891 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:11:29.891 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:11:29.892 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:11:29.892 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.892 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -3, 2021-05-12T09:11:29.892 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:11:29.892 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:11:29.893 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.893 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.893 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.893 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.894 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -2, 2021-05-12T09:11:29.894 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "default~hdd", 2021-05-12T09:11:29.894 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:11:29.894 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "root", 2021-05-12T09:11:29.895 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:11:29.895 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:11:29.895 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:11:29.895 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:11:29.895 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.896 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -4, 2021-05-12T09:11:29.896 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:11:29.896 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:11:29.896 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.897 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.897 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.897 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.897 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -3, 2021-05-12T09:11:29.897 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "smithi028", 2021-05-12T09:11:29.898 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:11:29.898 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "host", 2021-05-12T09:11:29.898 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:11:29.898 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:11:29.899 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:11:29.899 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:11:29.899 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.899 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:11:29.899 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:11:29.900 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:11:29.900 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.900 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.900 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:11:29.901 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:11:29.901 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 1 2021-05-12T09:11:29.901 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.901 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.901 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:11:29.902 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:11:29.902 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 2 2021-05-12T09:11:29.902 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.902 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.903 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.903 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.903 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -4, 2021-05-12T09:11:29.903 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "smithi028~hdd", 2021-05-12T09:11:29.903 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:11:29.904 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "host", 2021-05-12T09:11:29.904 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:11:29.904 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:11:29.905 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:11:29.905 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:11:29.905 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.905 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:11:29.905 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:11:29.906 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:11:29.906 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.906 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.906 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:11:29.907 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:11:29.907 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 1 2021-05-12T09:11:29.907 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.907 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.908 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:11:29.908 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:11:29.908 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 2 2021-05-12T09:11:29.908 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.908 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.909 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.909 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.909 INFO:tasks.workunit.client.0.smithi028.stdout: "rules": [ 2021-05-12T09:11:29.909 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.910 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_id": 0, 2021-05-12T09:11:29.910 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_name": "replicated_rule", 2021-05-12T09:11:29.910 INFO:tasks.workunit.client.0.smithi028.stdout: "ruleset": 0, 2021-05-12T09:11:29.910 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:11:29.910 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 1, 2021-05-12T09:11:29.911 INFO:tasks.workunit.client.0.smithi028.stdout: "max_size": 10, 2021-05-12T09:11:29.911 INFO:tasks.workunit.client.0.smithi028.stdout: "steps": [ 2021-05-12T09:11:29.911 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.911 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "take", 2021-05-12T09:11:29.912 INFO:tasks.workunit.client.0.smithi028.stdout: "item": -1, 2021-05-12T09:11:29.912 INFO:tasks.workunit.client.0.smithi028.stdout: "item_name": "default" 2021-05-12T09:11:29.912 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.912 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.913 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "choose_firstn", 2021-05-12T09:11:29.913 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 0, 2021-05-12T09:11:29.913 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "osd" 2021-05-12T09:11:29.913 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.913 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.914 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "emit" 2021-05-12T09:11:29.914 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.914 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:11:29.914 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.915 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.915 INFO:tasks.workunit.client.0.smithi028.stdout: "tunables": { 2021-05-12T09:11:29.915 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_local_tries": 0, 2021-05-12T09:11:29.915 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_local_fallback_tries": 0, 2021-05-12T09:11:29.916 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_total_tries": 50, 2021-05-12T09:11:29.916 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_descend_once": 1, 2021-05-12T09:11:29.916 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_vary_r": 1, 2021-05-12T09:11:29.916 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_stable": 1, 2021-05-12T09:11:29.917 INFO:tasks.workunit.client.0.smithi028.stdout: "straw_calc_version": 1, 2021-05-12T09:11:29.917 INFO:tasks.workunit.client.0.smithi028.stdout: "allowed_bucket_algs": 54, 2021-05-12T09:11:29.917 INFO:tasks.workunit.client.0.smithi028.stdout: "profile": "jewel", 2021-05-12T09:11:29.917 INFO:tasks.workunit.client.0.smithi028.stdout: "optimal_tunables": 1, 2021-05-12T09:11:29.917 INFO:tasks.workunit.client.0.smithi028.stdout: "legacy_tunables": 0, 2021-05-12T09:11:29.918 INFO:tasks.workunit.client.0.smithi028.stdout: "minimum_required_version": "jewel", 2021-05-12T09:11:29.918 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables": 1, 2021-05-12T09:11:29.918 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables2": 1, 2021-05-12T09:11:29.918 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v2_rules": 0, 2021-05-12T09:11:29.919 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables3": 1, 2021-05-12T09:11:29.919 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v3_rules": 0, 2021-05-12T09:11:29.919 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v4_buckets": 1, 2021-05-12T09:11:29.919 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables5": 1, 2021-05-12T09:11:29.919 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v5_rules": 0 2021-05-12T09:11:29.920 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.920 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_args": {} 2021-05-12T09:11:29.920 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.920 INFO:tasks.workunit.client.0.smithi028.stdout: "fsmap": { 2021-05-12T09:11:29.921 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 1, 2021-05-12T09:11:29.921 INFO:tasks.workunit.client.0.smithi028.stdout: "default_fscid": -1, 2021-05-12T09:11:29.921 INFO:tasks.workunit.client.0.smithi028.stdout: "compat": { 2021-05-12T09:11:29.921 INFO:tasks.workunit.client.0.smithi028.stdout: "compat": {}, 2021-05-12T09:11:29.921 INFO:tasks.workunit.client.0.smithi028.stdout: "ro_compat": {}, 2021-05-12T09:11:29.922 INFO:tasks.workunit.client.0.smithi028.stdout: "incompat": { 2021-05-12T09:11:29.922 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_1": "base v0.20", 2021-05-12T09:11:29.922 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_2": "client writeable ranges", 2021-05-12T09:11:29.922 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T09:11:29.923 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_4": "dir inode in separate object", 2021-05-12T09:11:29.923 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T09:11:29.923 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T09:11:29.923 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_8": "no anchor table", 2021-05-12T09:11:29.923 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_9": "file layout v2", 2021-05-12T09:11:29.924 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_10": "snaprealm v2" 2021-05-12T09:11:29.924 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.924 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.924 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_flags": { 2021-05-12T09:11:29.925 INFO:tasks.workunit.client.0.smithi028.stdout: "enable_multiple": true, 2021-05-12T09:11:29.925 INFO:tasks.workunit.client.0.smithi028.stdout: "ever_enabled_multiple": true 2021-05-12T09:11:29.925 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.925 INFO:tasks.workunit.client.0.smithi028.stdout: "standbys": [], 2021-05-12T09:11:29.926 INFO:tasks.workunit.client.0.smithi028.stdout: "filesystems": [] 2021-05-12T09:11:29.926 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.926 INFO:tasks.workunit.client.0.smithi028.stdout: "mdsmap_first_committed": 1, 2021-05-12T09:11:29.926 INFO:tasks.workunit.client.0.smithi028.stdout: "mdsmap_last_committed": 1, 2021-05-12T09:11:29.926 INFO:tasks.workunit.client.0.smithi028.stdout: "auth": { 2021-05-12T09:11:29.927 INFO:tasks.workunit.client.0.smithi028.stdout: "first_committed": 1, 2021-05-12T09:11:29.927 INFO:tasks.workunit.client.0.smithi028.stdout: "last_committed": 5, 2021-05-12T09:11:29.927 INFO:tasks.workunit.client.0.smithi028.stdout: "num_secrets": 3 2021-05-12T09:11:29.927 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.928 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 65, 2021-05-12T09:11:29.928 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_active": 65, 2021-05-12T09:11:29.928 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_unknown": 0, 2021-05-12T09:11:29.928 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osd": 3, 2021-05-12T09:11:29.929 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_sum": { 2021-05-12T09:11:29.929 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:11:29.929 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 507227, 2021-05-12T09:11:29.929 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 102, 2021-05-12T09:11:29.930 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:11:29.930 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 306, 2021-05-12T09:11:29.930 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:11:29.930 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:11:29.930 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 102, 2021-05-12T09:11:29.931 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:11:29.931 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:11:29.931 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 102, 2021-05-12T09:11:29.931 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:11:29.932 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 76, 2021-05-12T09:11:29.932 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 64, 2021-05-12T09:11:29.932 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 196, 2021-05-12T09:11:29.932 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 1211, 2021-05-12T09:11:29.932 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:11:29.933 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:11:29.933 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:11:29.933 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:11:29.933 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:11:29.934 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:11:29.934 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:11:29.934 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:11:29.934 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:11:29.935 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:11:29.935 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:11:29.935 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:11:29.935 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:11:29.935 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:11:29.936 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:11:29.936 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:11:29.936 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:11:29.936 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:11:29.937 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:11:29.937 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:11:29.937 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:11:29.937 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:11:29.938 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:11:29.938 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:11:29.938 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:11:29.938 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.939 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:11:29.939 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:11:29.939 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:11:29.939 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:11:29.940 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 0, 2021-05-12T09:11:29.940 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 0, 2021-05-12T09:11:29.940 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:11:29.940 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:11:29.940 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:11:29.941 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:11:29.941 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:11:29.941 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.941 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 1039, 2021-05-12T09:11:29.942 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 1039, 2021-05-12T09:11:29.942 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 130, 2021-05-12T09:11:29.942 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 130, 2021-05-12T09:11:29.942 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 0 2021-05-12T09:11:29.942 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.943 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_sum": { 2021-05-12T09:11:29.943 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 0, 2021-05-12T09:11:29.943 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 0, 2021-05-12T09:11:29.943 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pgs": 132, 2021-05-12T09:11:29.944 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osds": 3, 2021-05-12T09:11:29.944 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_osds": 3, 2021-05-12T09:11:29.944 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T09:11:29.944 INFO:tasks.workunit.client.0.smithi028.stdout: "kb": 314572800, 2021-05-12T09:11:29.945 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used": 24652, 2021-05-12T09:11:29.945 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_data": 3700, 2021-05-12T09:11:29.945 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_omap": 2, 2021-05-12T09:11:29.945 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_meta": 20925, 2021-05-12T09:11:29.945 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_avail": 314548148, 2021-05-12T09:11:29.946 INFO:tasks.workunit.client.0.smithi028.stdout: "statfs": { 2021-05-12T09:11:29.946 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 322122547200, 2021-05-12T09:11:29.946 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 322097303552, 2021-05-12T09:11:29.946 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:11:29.947 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 3788800, 2021-05-12T09:11:29.947 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1929547, 2021-05-12T09:11:29.947 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:11:29.947 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:11:29.948 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:11:29.948 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 2160, 2021-05-12T09:11:29.948 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 21428112 2021-05-12T09:11:29.948 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.949 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_peers": [], 2021-05-12T09:11:29.949 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_trim_queue_len": 0, 2021-05-12T09:11:29.949 INFO:tasks.workunit.client.0.smithi028.stdout: "num_snap_trimming": 0, 2021-05-12T09:11:29.949 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shards_repaired": 0, 2021-05-12T09:11:29.949 INFO:tasks.workunit.client.0.smithi028.stdout: "op_queue_age_hist": { 2021-05-12T09:11:29.950 INFO:tasks.workunit.client.0.smithi028.stdout: "histogram": [], 2021-05-12T09:11:29.950 INFO:tasks.workunit.client.0.smithi028.stdout: "upper_bound": 1 2021-05-12T09:11:29.950 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.950 INFO:tasks.workunit.client.0.smithi028.stdout: "perf_stat": { 2021-05-12T09:11:29.951 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ms": 56, 2021-05-12T09:11:29.951 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ms": 56, 2021-05-12T09:11:29.951 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ns": 56000000, 2021-05-12T09:11:29.951 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ns": 56000000 2021-05-12T09:11:29.951 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.952 INFO:tasks.workunit.client.0.smithi028.stdout: "alerts": [], 2021-05-12T09:11:29.952 INFO:tasks.workunit.client.0.smithi028.stdout: "network_ping_times": [] 2021-05-12T09:11:29.952 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.952 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_sum_by_class": { 2021-05-12T09:11:29.953 INFO:tasks.workunit.client.0.smithi028.stdout: "hdd": { 2021-05-12T09:11:29.953 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 0, 2021-05-12T09:11:29.953 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 0, 2021-05-12T09:11:29.953 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pgs": 132, 2021-05-12T09:11:29.954 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osds": 3, 2021-05-12T09:11:29.954 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_osds": 3, 2021-05-12T09:11:29.954 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T09:11:29.954 INFO:tasks.workunit.client.0.smithi028.stdout: "kb": 314572800, 2021-05-12T09:11:29.954 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used": 24652, 2021-05-12T09:11:29.955 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_data": 3700, 2021-05-12T09:11:29.955 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_omap": 2, 2021-05-12T09:11:29.955 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_meta": 20925, 2021-05-12T09:11:29.955 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_avail": 314548148, 2021-05-12T09:11:29.956 INFO:tasks.workunit.client.0.smithi028.stdout: "statfs": { 2021-05-12T09:11:29.956 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 322122547200, 2021-05-12T09:11:29.956 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 322097303552, 2021-05-12T09:11:29.956 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:11:29.957 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 3788800, 2021-05-12T09:11:29.957 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1929547, 2021-05-12T09:11:29.957 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:11:29.957 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:11:29.957 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:11:29.958 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 2160, 2021-05-12T09:11:29.958 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 21428112 2021-05-12T09:11:29.958 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.958 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_peers": [], 2021-05-12T09:11:29.959 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_trim_queue_len": 0, 2021-05-12T09:11:29.959 INFO:tasks.workunit.client.0.smithi028.stdout: "num_snap_trimming": 0, 2021-05-12T09:11:29.959 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shards_repaired": 0, 2021-05-12T09:11:29.959 INFO:tasks.workunit.client.0.smithi028.stdout: "op_queue_age_hist": { 2021-05-12T09:11:29.960 INFO:tasks.workunit.client.0.smithi028.stdout: "histogram": [], 2021-05-12T09:11:29.960 INFO:tasks.workunit.client.0.smithi028.stdout: "upper_bound": 1 2021-05-12T09:11:29.960 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.960 INFO:tasks.workunit.client.0.smithi028.stdout: "perf_stat": { 2021-05-12T09:11:29.961 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ms": 56, 2021-05-12T09:11:29.961 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ms": 56, 2021-05-12T09:11:29.961 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ns": 56000000, 2021-05-12T09:11:29.961 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ns": 56000000 2021-05-12T09:11:29.961 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.962 INFO:tasks.workunit.client.0.smithi028.stdout: "alerts": [], 2021-05-12T09:11:29.962 INFO:tasks.workunit.client.0.smithi028.stdout: "network_ping_times": [] 2021-05-12T09:11:29.962 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.962 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.963 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_stats": [ 2021-05-12T09:11:29.963 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.963 INFO:tasks.workunit.client.0.smithi028.stdout: "poolid": 2, 2021-05-12T09:11:29.963 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 64, 2021-05-12T09:11:29.964 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:11:29.964 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 47947, 2021-05-12T09:11:29.964 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 100, 2021-05-12T09:11:29.964 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:11:29.964 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 300, 2021-05-12T09:11:29.965 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:11:29.965 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:11:29.965 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 100, 2021-05-12T09:11:29.965 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:11:29.966 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:11:29.966 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 100, 2021-05-12T09:11:29.966 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:11:29.966 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 0, 2021-05-12T09:11:29.967 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 0, 2021-05-12T09:11:29.967 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 101, 2021-05-12T09:11:29.967 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 101, 2021-05-12T09:11:29.967 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:11:29.967 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:11:29.968 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:11:29.968 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:11:29.968 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:11:29.968 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:11:29.969 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:11:29.969 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:11:29.969 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:11:29.969 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:11:29.969 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:11:29.970 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:11:29.970 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:11:29.970 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:11:29.970 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:11:29.971 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:11:29.971 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:11:29.971 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:11:29.971 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:11:29.972 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:11:29.972 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:11:29.972 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:11:29.972 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:11:29.972 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:11:29.973 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:11:29.973 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.973 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:11:29.973 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:11:29.974 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:11:29.974 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:11:29.974 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 1220608, 2021-05-12T09:11:29.975 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 142522, 2021-05-12T09:11:29.975 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:11:29.975 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:11:29.975 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:11:29.975 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:11:29.976 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:11:29.976 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.976 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 1025, 2021-05-12T09:11:29.976 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 1025, 2021-05-12T09:11:29.977 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 128, 2021-05-12T09:11:29.977 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 128, 2021-05-12T09:11:29.977 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 3 2021-05-12T09:11:29.977 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.977 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.978 INFO:tasks.workunit.client.0.smithi028.stdout: "poolid": 1, 2021-05-12T09:11:29.978 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 1, 2021-05-12T09:11:29.978 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:11:29.978 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 459280, 2021-05-12T09:11:29.979 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 2, 2021-05-12T09:11:29.979 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:11:29.979 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 6, 2021-05-12T09:11:29.979 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:11:29.979 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:11:29.980 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 2, 2021-05-12T09:11:29.980 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:11:29.980 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:11:29.980 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 2, 2021-05-12T09:11:29.981 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:11:29.981 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 76, 2021-05-12T09:11:29.981 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 64, 2021-05-12T09:11:29.981 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 95, 2021-05-12T09:11:29.982 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 1110, 2021-05-12T09:11:29.982 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:11:29.982 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:11:29.983 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:11:29.983 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:11:29.983 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:11:29.983 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:11:29.987 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:11:29.988 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:11:29.988 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:11:29.988 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:11:29.988 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:11:29.989 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:11:29.989 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:11:29.989 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:11:29.989 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:11:29.989 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:11:29.990 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:11:29.990 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:11:29.990 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:11:29.990 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:11:29.991 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:11:29.991 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:11:29.991 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:11:29.991 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:11:29.991 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:11:29.992 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.992 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:11:29.992 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:11:29.992 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:11:29.993 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:11:29.993 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 1388544, 2021-05-12T09:11:29.993 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1377840, 2021-05-12T09:11:29.993 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:11:29.993 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:11:29.994 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:11:29.994 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:11:29.994 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:11:29.994 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.994 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 14, 2021-05-12T09:11:29.995 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 14, 2021-05-12T09:11:29.995 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 2, 2021-05-12T09:11:29.995 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 2, 2021-05-12T09:11:29.995 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 3 2021-05-12T09:11:29.996 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.996 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.996 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_stats": [ 2021-05-12T09:11:29.996 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.996 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:11:29.997 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 184683593748 2021-05-12T09:11:29.997 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.997 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.997 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:11:29.997 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 193273528339 2021-05-12T09:11:29.998 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:29.998 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.998 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:11:29.998 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 64424509455 2021-05-12T09:11:29.999 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:29.999 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:29.999 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_by_state": [ 2021-05-12T09:11:29.999 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:29.999 INFO:tasks.workunit.client.0.smithi028.stdout: "state": "active+undersized+degraded", 2021-05-12T09:11:30.000 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 65 2021-05-12T09:11:30.000 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:30.000 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:30.000 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_by_osd": [ 2021-05-12T09:11:30.001 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:30.001 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:11:30.001 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:11:30.001 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 0, 2021-05-12T09:11:30.001 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:11:30.002 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:30.002 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:30.002 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:11:30.002 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 65, 2021-05-12T09:11:30.002 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 65, 2021-05-12T09:11:30.003 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:11:30.003 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:30.003 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:30.003 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:11:30.004 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:11:30.004 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 65, 2021-05-12T09:11:30.004 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:11:30.004 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:30.004 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:30.005 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": [ 2021-05-12T09:11:30.005 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:30.005 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 1, 2021-05-12T09:11:30.005 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": {} 2021-05-12T09:11:30.005 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:30.006 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:11:30.006 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 2, 2021-05-12T09:11:30.006 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": {} 2021-05-12T09:11:30.006 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:30.007 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:11:30.007 INFO:tasks.workunit.client.0.smithi028.stdout: "servicemap": { 2021-05-12T09:11:30.007 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 7, 2021-05-12T09:11:30.007 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:11:16.058758+0000", 2021-05-12T09:11:30.007 INFO:tasks.workunit.client.0.smithi028.stdout: "services": {} 2021-05-12T09:11:30.008 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:11:30.008 INFO:tasks.workunit.client.0.smithi028.stdout: "paxos": { 2021-05-12T09:11:30.008 INFO:tasks.workunit.client.0.smithi028.stdout: "first_committed": 1, 2021-05-12T09:11:30.008 INFO:tasks.workunit.client.0.smithi028.stdout: "last_committed": 478, 2021-05-12T09:11:30.008 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pn": 0, 2021-05-12T09:11:30.009 INFO:tasks.workunit.client.0.smithi028.stdout: "accepted_pn": 0 2021-05-12T09:11:30.009 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:11:30.009 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:11:30.010 INFO:tasks.workunit.client.0.smithi028.stderr:report 1029286813 2021-05-12T09:11:30.010 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T09:11:30.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2021-05-12T09:11:30.011 INFO:tasks.workunit.client.0.smithi028.stdout:delay recovery 2021-05-12T09:11:30.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2021-05-12T09:11:30.031 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:11:30.031 INFO:tasks.workunit.client.0.smithi028.stdout:2.3f 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325151+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.031 INFO:tasks.workunit.client.0.smithi028.stdout:2.3e 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.352066+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.031 INFO:tasks.workunit.client.0.smithi028.stdout:2.3d 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324201+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.032 INFO:tasks.workunit.client.0.smithi028.stdout:2.3c 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324071+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.032 INFO:tasks.workunit.client.0.smithi028.stdout:2.3b 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323529+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.032 INFO:tasks.workunit.client.0.smithi028.stdout:2.3a 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326590+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.032 INFO:tasks.workunit.client.0.smithi028.stdout:2.39 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318815+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.033 INFO:tasks.workunit.client.0.smithi028.stdout:2.38 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320936+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.033 INFO:tasks.workunit.client.0.smithi028.stdout:2.37 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321488+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.033 INFO:tasks.workunit.client.0.smithi028.stdout:2.36 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326290+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.033 INFO:tasks.workunit.client.0.smithi028.stdout:2.35 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319374+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.033 INFO:tasks.workunit.client.0.smithi028.stdout:2.34 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321682+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.034 INFO:tasks.workunit.client.0.smithi028.stdout:2.33 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326133+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.034 INFO:tasks.workunit.client.0.smithi028.stdout:2.32 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326137+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.034 INFO:tasks.workunit.client.0.smithi028.stdout:2.31 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319432+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.034 INFO:tasks.workunit.client.0.smithi028.stdout:2.30 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320564+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.035 INFO:tasks.workunit.client.0.smithi028.stdout:2.2f 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.327561+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.035 INFO:tasks.workunit.client.0.smithi028.stdout:2.2e 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325408+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.035 INFO:tasks.workunit.client.0.smithi028.stdout:2.2d 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318417+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.035 INFO:tasks.workunit.client.0.smithi028.stdout:2.2c 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320637+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.035 INFO:tasks.workunit.client.0.smithi028.stdout:2.2b 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326283+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.036 INFO:tasks.workunit.client.0.smithi028.stdout:2.2a 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.327146+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.036 INFO:tasks.workunit.client.0.smithi028.stdout:2.29 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320138+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.036 INFO:tasks.workunit.client.0.smithi028.stdout:2.28 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320304+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.036 INFO:tasks.workunit.client.0.smithi028.stdout:2.27 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325633+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.037 INFO:tasks.workunit.client.0.smithi028.stdout:2.26 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325591+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.037 INFO:tasks.workunit.client.0.smithi028.stdout:2.25 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326279+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.037 INFO:tasks.workunit.client.0.smithi028.stdout:2.24 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323195+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.037 INFO:tasks.workunit.client.0.smithi028.stdout:2.23 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325469+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.037 INFO:tasks.workunit.client.0.smithi028.stdout:2.e 2 0 2 0 0 402 0 0 17 17 active+undersized+degraded 2021-05-12T09:10:17.320245+0000 51'101 51:125 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.038 INFO:tasks.workunit.client.0.smithi028.stdout:2.d 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321136+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.038 INFO:tasks.workunit.client.0.smithi028.stdout:2.c 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.322090+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.038 INFO:tasks.workunit.client.0.smithi028.stdout:2.b 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319604+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.038 INFO:tasks.workunit.client.0.smithi028.stdout:2.a 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319929+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.039 INFO:tasks.workunit.client.0.smithi028.stdout:2.9 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323963+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.039 INFO:tasks.workunit.client.0.smithi028.stdout:2.8 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323896+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.039 INFO:tasks.workunit.client.0.smithi028.stdout:2.7 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325446+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.039 INFO:tasks.workunit.client.0.smithi028.stdout:2.6 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.322282+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.040 INFO:tasks.workunit.client.0.smithi028.stdout:2.5 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318856+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.040 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321112+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.040 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:10:17.320695+0000 33'62 51:116 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:20.116394+0000 0'0 2021-05-12T09:09:20.116394+0000 0 2021-05-12T09:11:30.040 INFO:tasks.workunit.client.0.smithi028.stdout:2.3 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321973+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.040 INFO:tasks.workunit.client.0.smithi028.stdout:2.1 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321360+0000 33'100 51:123 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.041 INFO:tasks.workunit.client.0.smithi028.stdout:2.2 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323329+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.041 INFO:tasks.workunit.client.0.smithi028.stdout:2.4 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321691+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.041 INFO:tasks.workunit.client.0.smithi028.stdout:2.f 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.220557+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.041 INFO:tasks.workunit.client.0.smithi028.stdout:2.10 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321851+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.042 INFO:tasks.workunit.client.0.smithi028.stdout:2.11 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324153+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.042 INFO:tasks.workunit.client.0.smithi028.stdout:2.12 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320164+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.042 INFO:tasks.workunit.client.0.smithi028.stdout:2.13 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318763+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.042 INFO:tasks.workunit.client.0.smithi028.stdout:2.14 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319539+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.042 INFO:tasks.workunit.client.0.smithi028.stdout:2.15 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318618+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.043 INFO:tasks.workunit.client.0.smithi028.stdout:2.16 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321253+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.043 INFO:tasks.workunit.client.0.smithi028.stdout:2.17 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.220474+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.043 INFO:tasks.workunit.client.0.smithi028.stdout:2.18 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319857+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.043 INFO:tasks.workunit.client.0.smithi028.stdout:2.19 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321903+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.044 INFO:tasks.workunit.client.0.smithi028.stdout:2.1a 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320526+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.044 INFO:tasks.workunit.client.0.smithi028.stdout:2.1b 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318500+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.044 INFO:tasks.workunit.client.0.smithi028.stdout:2.1c 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319781+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.044 INFO:tasks.workunit.client.0.smithi028.stdout:2.1d 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318542+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.044 INFO:tasks.workunit.client.0.smithi028.stdout:2.1e 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323197+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.045 INFO:tasks.workunit.client.0.smithi028.stdout:2.1f 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323951+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.045 INFO:tasks.workunit.client.0.smithi028.stdout:2.20 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323744+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.045 INFO:tasks.workunit.client.0.smithi028.stdout:2.21 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323484+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.045 INFO:tasks.workunit.client.0.smithi028.stdout:2.22 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325216+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.045 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:11:30.046 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:11:30.048 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:11:30.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2021-05-12T09:11:30.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2021-05-12T09:11:30.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T09:11:30.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T09:11:30.049 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:11:30.050 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:11:30.050 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:11:30.050 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.0.asok 2021-05-12T09:11:30.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2021-05-12T09:11:30.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.0.asok set_recovery_delay 100000 2021-05-12T09:11:30.143 INFO:tasks.workunit.client.0.smithi028.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T09:11:30.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2021-05-12T09:11:30.153 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.1 2021-05-12T09:11:30.153 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T09:11:30.153 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T09:11:30.153 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:11:30.154 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:11:30.154 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:11:30.154 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.1.asok 2021-05-12T09:11:30.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2021-05-12T09:11:30.155 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.1.asok set_recovery_delay 100000 2021-05-12T09:11:30.253 INFO:tasks.workunit.client.0.smithi028.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T09:11:30.261 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 2' 2021-05-12T09:11:30.262 INFO:tasks.workunit.client.0.smithi028.stdout:revive divergent 2 2021-05-12T09:11:30.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2021-05-12T09:11:30.626 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:11:30.626 INFO:tasks.workunit.client.0.smithi028.stdout:2.3f 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325151+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.626 INFO:tasks.workunit.client.0.smithi028.stdout:2.3e 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.352066+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.627 INFO:tasks.workunit.client.0.smithi028.stdout:2.3d 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324201+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.627 INFO:tasks.workunit.client.0.smithi028.stdout:2.3c 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324071+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.627 INFO:tasks.workunit.client.0.smithi028.stdout:2.3b 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323529+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.627 INFO:tasks.workunit.client.0.smithi028.stdout:2.3a 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326590+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.628 INFO:tasks.workunit.client.0.smithi028.stdout:2.39 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318815+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.628 INFO:tasks.workunit.client.0.smithi028.stdout:2.38 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320936+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.628 INFO:tasks.workunit.client.0.smithi028.stdout:2.37 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321488+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.628 INFO:tasks.workunit.client.0.smithi028.stdout:2.36 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326290+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.628 INFO:tasks.workunit.client.0.smithi028.stdout:2.35 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319374+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.629 INFO:tasks.workunit.client.0.smithi028.stdout:2.34 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321682+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.629 INFO:tasks.workunit.client.0.smithi028.stdout:2.33 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326133+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.629 INFO:tasks.workunit.client.0.smithi028.stdout:2.32 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326137+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.629 INFO:tasks.workunit.client.0.smithi028.stdout:2.31 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319432+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.630 INFO:tasks.workunit.client.0.smithi028.stdout:2.30 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320564+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.630 INFO:tasks.workunit.client.0.smithi028.stdout:2.2f 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.327561+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.630 INFO:tasks.workunit.client.0.smithi028.stdout:2.2e 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325408+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.630 INFO:tasks.workunit.client.0.smithi028.stdout:2.2d 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318417+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.630 INFO:tasks.workunit.client.0.smithi028.stdout:2.2c 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320637+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.631 INFO:tasks.workunit.client.0.smithi028.stdout:2.2b 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326283+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.631 INFO:tasks.workunit.client.0.smithi028.stdout:2.2a 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.327146+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.631 INFO:tasks.workunit.client.0.smithi028.stdout:2.29 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320138+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.631 INFO:tasks.workunit.client.0.smithi028.stdout:2.28 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320304+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.632 INFO:tasks.workunit.client.0.smithi028.stdout:2.27 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325633+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.632 INFO:tasks.workunit.client.0.smithi028.stdout:2.26 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325591+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.632 INFO:tasks.workunit.client.0.smithi028.stdout:2.25 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326279+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.632 INFO:tasks.workunit.client.0.smithi028.stdout:2.24 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323195+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.633 INFO:tasks.workunit.client.0.smithi028.stdout:2.23 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325469+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.633 INFO:tasks.workunit.client.0.smithi028.stdout:2.e 2 0 2 0 0 402 0 0 17 17 active+undersized+degraded 2021-05-12T09:10:17.320245+0000 51'101 51:125 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.633 INFO:tasks.workunit.client.0.smithi028.stdout:2.d 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321136+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.633 INFO:tasks.workunit.client.0.smithi028.stdout:2.c 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.322090+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.634 INFO:tasks.workunit.client.0.smithi028.stdout:2.b 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319604+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.634 INFO:tasks.workunit.client.0.smithi028.stdout:2.a 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319929+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.634 INFO:tasks.workunit.client.0.smithi028.stdout:2.9 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323963+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.634 INFO:tasks.workunit.client.0.smithi028.stdout:2.8 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323896+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.634 INFO:tasks.workunit.client.0.smithi028.stdout:2.7 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325446+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.635 INFO:tasks.workunit.client.0.smithi028.stdout:2.6 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.322282+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.635 INFO:tasks.workunit.client.0.smithi028.stdout:2.5 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318856+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.635 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321112+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.635 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:10:17.320695+0000 33'62 51:116 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:20.116394+0000 0'0 2021-05-12T09:09:20.116394+0000 0 2021-05-12T09:11:30.636 INFO:tasks.workunit.client.0.smithi028.stdout:2.3 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321973+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.636 INFO:tasks.workunit.client.0.smithi028.stdout:2.1 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321360+0000 33'100 51:123 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.636 INFO:tasks.workunit.client.0.smithi028.stdout:2.2 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323329+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.636 INFO:tasks.workunit.client.0.smithi028.stdout:2.4 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321691+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.636 INFO:tasks.workunit.client.0.smithi028.stdout:2.f 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.220557+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.637 INFO:tasks.workunit.client.0.smithi028.stdout:2.10 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321851+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.637 INFO:tasks.workunit.client.0.smithi028.stdout:2.11 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324153+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.637 INFO:tasks.workunit.client.0.smithi028.stdout:2.12 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320164+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.637 INFO:tasks.workunit.client.0.smithi028.stdout:2.13 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318763+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.638 INFO:tasks.workunit.client.0.smithi028.stdout:2.14 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319539+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.638 INFO:tasks.workunit.client.0.smithi028.stdout:2.15 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318618+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.638 INFO:tasks.workunit.client.0.smithi028.stdout:2.16 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321253+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.638 INFO:tasks.workunit.client.0.smithi028.stdout:2.17 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.220474+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.638 INFO:tasks.workunit.client.0.smithi028.stdout:2.18 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319857+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.639 INFO:tasks.workunit.client.0.smithi028.stdout:2.19 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321903+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.639 INFO:tasks.workunit.client.0.smithi028.stdout:2.1a 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320526+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.639 INFO:tasks.workunit.client.0.smithi028.stdout:2.1b 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318500+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.639 INFO:tasks.workunit.client.0.smithi028.stdout:2.1c 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319781+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.640 INFO:tasks.workunit.client.0.smithi028.stdout:2.1d 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318542+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.640 INFO:tasks.workunit.client.0.smithi028.stdout:2.1e 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323197+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.640 INFO:tasks.workunit.client.0.smithi028.stdout:2.1f 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323951+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.640 INFO:tasks.workunit.client.0.smithi028.stdout:2.20 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323744+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.641 INFO:tasks.workunit.client.0.smithi028.stdout:2.21 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323484+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.641 INFO:tasks.workunit.client.0.smithi028.stdout:2.22 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325216+0000 33'100 51:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:30.641 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:11:30.641 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:11:30.642 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:11:30.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2021-05-12T09:11:31.258 INFO:tasks.workunit.client.0.smithi028.stderr:noup is set 2021-05-12T09:11:31.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 2 2021-05-12T09:11:31.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:11:31.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:11:31.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T09:11:31.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:11:31.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T09:11:31.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:11:31.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:11:31.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:11:31.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:11:31.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T09:11:31.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T09:11:31.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:11:31.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:11:31.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:11:31.275 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:11:31.275 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:11:31.275 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:11:31.275 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:11:31.275 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:11:31.276 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:11:31.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:11:31.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:11:31.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:11:31.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:11:31.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:11:31.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:11:31.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:11:31.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:11:31.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:11:31.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:11:31.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T09:11:31.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T09:11:31.279 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:11:31.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:11:31.280 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T09:11:31.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T09:11:31.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:11:31.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:11:31.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:11:31.300 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:11:31.299+0000 7f4f1cdcf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:11:31.306 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:11:31.305+0000 7f4f1cdcf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:11:31.307 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:11:31.306+0000 7f4f1cdcf0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:11:31.664 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: return 0 2021-05-12T09:11:31.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2021-05-12T09:11:31.823 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:11:31.822+0000 7f4f1cdcf0c0 -1 Falling back to public interface 2021-05-12T09:11:34.550 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:11:34.549+0000 7f4f1cdcf0c0 -1 osd.2 36 log_to_monitors {default=true} 2021-05-12T09:11:36.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2021-05-12T09:11:36.666 INFO:tasks.workunit.client.0.smithi028.stdout:delay recovery divergent 2021-05-12T09:11:36.666 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2021-05-12T09:11:37.045 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:11:37.045 INFO:tasks.workunit.client.0.smithi028.stdout:2.3f 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325151+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.045 INFO:tasks.workunit.client.0.smithi028.stdout:2.3e 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.352066+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.045 INFO:tasks.workunit.client.0.smithi028.stdout:2.3d 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324201+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.046 INFO:tasks.workunit.client.0.smithi028.stdout:2.3c 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324071+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.046 INFO:tasks.workunit.client.0.smithi028.stdout:2.3b 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323529+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.046 INFO:tasks.workunit.client.0.smithi028.stdout:2.3a 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326590+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.046 INFO:tasks.workunit.client.0.smithi028.stdout:2.39 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318815+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.047 INFO:tasks.workunit.client.0.smithi028.stdout:2.38 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320936+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.047 INFO:tasks.workunit.client.0.smithi028.stdout:2.37 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321488+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.047 INFO:tasks.workunit.client.0.smithi028.stdout:2.36 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326290+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.047 INFO:tasks.workunit.client.0.smithi028.stdout:2.35 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319374+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.048 INFO:tasks.workunit.client.0.smithi028.stdout:2.34 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321682+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.048 INFO:tasks.workunit.client.0.smithi028.stdout:2.33 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326133+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.048 INFO:tasks.workunit.client.0.smithi028.stdout:2.32 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326137+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.048 INFO:tasks.workunit.client.0.smithi028.stdout:2.31 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319432+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.048 INFO:tasks.workunit.client.0.smithi028.stdout:2.30 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320564+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.049 INFO:tasks.workunit.client.0.smithi028.stdout:2.2f 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.327561+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.049 INFO:tasks.workunit.client.0.smithi028.stdout:2.2e 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325408+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.049 INFO:tasks.workunit.client.0.smithi028.stdout:2.2d 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318417+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.049 INFO:tasks.workunit.client.0.smithi028.stdout:2.2c 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320637+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.050 INFO:tasks.workunit.client.0.smithi028.stdout:2.2b 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326283+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.050 INFO:tasks.workunit.client.0.smithi028.stdout:2.2a 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.327146+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.050 INFO:tasks.workunit.client.0.smithi028.stdout:2.29 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320138+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.051 INFO:tasks.workunit.client.0.smithi028.stdout:2.28 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320304+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.051 INFO:tasks.workunit.client.0.smithi028.stdout:2.27 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325633+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.051 INFO:tasks.workunit.client.0.smithi028.stdout:2.26 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325591+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.051 INFO:tasks.workunit.client.0.smithi028.stdout:2.25 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326279+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.052 INFO:tasks.workunit.client.0.smithi028.stdout:2.24 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323195+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.052 INFO:tasks.workunit.client.0.smithi028.stdout:2.23 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325469+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.052 INFO:tasks.workunit.client.0.smithi028.stdout:2.e 2 0 2 0 0 402 0 0 17 17 active+undersized+degraded 2021-05-12T09:10:17.320245+0000 51'101 53:128 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.052 INFO:tasks.workunit.client.0.smithi028.stdout:2.d 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321136+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.052 INFO:tasks.workunit.client.0.smithi028.stdout:2.c 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.322090+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.053 INFO:tasks.workunit.client.0.smithi028.stdout:2.b 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319604+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.053 INFO:tasks.workunit.client.0.smithi028.stdout:2.a 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319929+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.053 INFO:tasks.workunit.client.0.smithi028.stdout:2.9 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323963+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.053 INFO:tasks.workunit.client.0.smithi028.stdout:2.8 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323896+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.054 INFO:tasks.workunit.client.0.smithi028.stdout:2.7 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325446+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.054 INFO:tasks.workunit.client.0.smithi028.stdout:2.6 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.322282+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.054 INFO:tasks.workunit.client.0.smithi028.stdout:2.5 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318856+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.054 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321112+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.054 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:10:17.320695+0000 33'62 53:119 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:20.116394+0000 0'0 2021-05-12T09:09:20.116394+0000 0 2021-05-12T09:11:37.055 INFO:tasks.workunit.client.0.smithi028.stdout:2.3 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321973+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.055 INFO:tasks.workunit.client.0.smithi028.stdout:2.1 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321360+0000 33'100 53:126 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.055 INFO:tasks.workunit.client.0.smithi028.stdout:2.2 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323329+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.055 INFO:tasks.workunit.client.0.smithi028.stdout:2.4 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321691+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.056 INFO:tasks.workunit.client.0.smithi028.stdout:2.f 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.220557+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.056 INFO:tasks.workunit.client.0.smithi028.stdout:2.10 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321851+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.056 INFO:tasks.workunit.client.0.smithi028.stdout:2.11 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324153+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.056 INFO:tasks.workunit.client.0.smithi028.stdout:2.12 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320164+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.057 INFO:tasks.workunit.client.0.smithi028.stdout:2.13 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318763+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.057 INFO:tasks.workunit.client.0.smithi028.stdout:2.14 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319539+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.057 INFO:tasks.workunit.client.0.smithi028.stdout:2.15 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318618+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.057 INFO:tasks.workunit.client.0.smithi028.stdout:2.16 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321253+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.057 INFO:tasks.workunit.client.0.smithi028.stdout:2.17 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.220474+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.058 INFO:tasks.workunit.client.0.smithi028.stdout:2.18 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319857+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.058 INFO:tasks.workunit.client.0.smithi028.stdout:2.19 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321903+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.058 INFO:tasks.workunit.client.0.smithi028.stdout:2.1a 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320526+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.058 INFO:tasks.workunit.client.0.smithi028.stdout:2.1b 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318500+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.059 INFO:tasks.workunit.client.0.smithi028.stdout:2.1c 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319781+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.059 INFO:tasks.workunit.client.0.smithi028.stdout:2.1d 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318542+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.059 INFO:tasks.workunit.client.0.smithi028.stdout:2.1e 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323197+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.059 INFO:tasks.workunit.client.0.smithi028.stdout:2.1f 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323951+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.059 INFO:tasks.workunit.client.0.smithi028.stdout:2.20 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323744+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.060 INFO:tasks.workunit.client.0.smithi028.stdout:2.21 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323484+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.060 INFO:tasks.workunit.client.0.smithi028.stdout:2.22 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325216+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:37.060 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:11:37.060 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:11:37.062 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:11:37.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.2 2021-05-12T09:11:37.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T09:11:37.063 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T09:11:37.063 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:11:37.063 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:11:37.063 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:11:37.064 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.2.asok 2021-05-12T09:11:37.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2021-05-12T09:11:37.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.2.asok set_recovery_delay 100000 2021-05-12T09:11:37.157 INFO:tasks.workunit.client.0.smithi028.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T09:11:37.165 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2021-05-12T09:11:38.255 INFO:tasks.workunit.client.0.smithi028.stderr:noup is unset 2021-05-12T09:11:38.265 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2021-05-12T09:11:38.266 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:11:38.266 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:11:38.266 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:11:38.266 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:11:38.267 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:11:38.267 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:11:38.267 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:11:38.268 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:11:38.268 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:11:38.716 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 43 up_thru 48 down_at 37 last_clean_interval [5,33) [v2:127.0.0.1:6802/63042,v1:127.0.0.1:6803/63042] [v2:127.0.0.1:6804/63042,v1:127.0.0.1:6805/63042] exists,up 18cc91da-f8e6-4da2-beb3-4002e77c4ec1 2021-05-12T09:11:38.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:11:38.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:11:38.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:11:38.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2021-05-12T09:11:38.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:11:38.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:11:38.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:11:38.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:11:38.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:11:38.720 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:11:38.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:11:38.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:11:38.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:11:39.118 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 45 up_thru 56 down_at 38 last_clean_interval [10,33) [v2:127.0.0.1:6810/63599,v1:127.0.0.1:6811/63599] [v2:127.0.0.1:6812/63599,v1:127.0.0.1:6813/63599] exists,up 602d76d3-2336-45c7-b45a-9e2846468b99 2021-05-12T09:11:39.119 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:11:39.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:11:39.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:11:39.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2021-05-12T09:11:39.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:11:39.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:11:39.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:11:39.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:11:39.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:11:39.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:11:39.122 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:11:39.122 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:11:39.123 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:11:39.534 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 56 up_thru 56 down_at 39 last_clean_interval [15,36) [v2:127.0.0.1:6818/66289,v1:127.0.0.1:6819/66289] [v2:127.0.0.1:6820/66289,v1:127.0.0.1:6821/66289] exists,up cabac835-d2f0-45a7-ba67-30ece6d00d83 2021-05-12T09:11:39.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:11:39.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:11:39.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:11:39.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2021-05-12T09:11:40.018 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:11:40.018 INFO:tasks.workunit.client.0.smithi028.stdout:2.3f 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325151+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.018 INFO:tasks.workunit.client.0.smithi028.stdout:2.3e 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.352066+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.019 INFO:tasks.workunit.client.0.smithi028.stdout:2.3d 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324201+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.021 INFO:tasks.workunit.client.0.smithi028.stdout:2.3c 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324071+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.021 INFO:tasks.workunit.client.0.smithi028.stdout:2.3b 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323529+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.022 INFO:tasks.workunit.client.0.smithi028.stdout:2.3a 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326590+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.022 INFO:tasks.workunit.client.0.smithi028.stdout:2.39 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318815+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.022 INFO:tasks.workunit.client.0.smithi028.stdout:2.38 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320936+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.022 INFO:tasks.workunit.client.0.smithi028.stdout:2.37 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321488+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.023 INFO:tasks.workunit.client.0.smithi028.stdout:2.36 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326290+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.023 INFO:tasks.workunit.client.0.smithi028.stdout:2.35 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319374+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.023 INFO:tasks.workunit.client.0.smithi028.stdout:2.34 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321682+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.023 INFO:tasks.workunit.client.0.smithi028.stdout:2.33 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326133+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.023 INFO:tasks.workunit.client.0.smithi028.stdout:2.32 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326137+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.024 INFO:tasks.workunit.client.0.smithi028.stdout:2.31 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319432+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.024 INFO:tasks.workunit.client.0.smithi028.stdout:2.30 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320564+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.024 INFO:tasks.workunit.client.0.smithi028.stdout:2.2f 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.327561+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.024 INFO:tasks.workunit.client.0.smithi028.stdout:2.2e 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325408+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.025 INFO:tasks.workunit.client.0.smithi028.stdout:2.2d 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318417+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.025 INFO:tasks.workunit.client.0.smithi028.stdout:2.2c 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320637+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.026 INFO:tasks.workunit.client.0.smithi028.stdout:2.2b 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326283+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.026 INFO:tasks.workunit.client.0.smithi028.stdout:2.2a 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.327146+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.027 INFO:tasks.workunit.client.0.smithi028.stdout:2.29 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320138+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.028 INFO:tasks.workunit.client.0.smithi028.stdout:2.28 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320304+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.029 INFO:tasks.workunit.client.0.smithi028.stdout:2.27 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325633+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.029 INFO:tasks.workunit.client.0.smithi028.stdout:2.26 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325591+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.029 INFO:tasks.workunit.client.0.smithi028.stdout:2.25 1 0 1 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.326279+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.029 INFO:tasks.workunit.client.0.smithi028.stdout:2.24 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323195+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.030 INFO:tasks.workunit.client.0.smithi028.stdout:2.23 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325469+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.030 INFO:tasks.workunit.client.0.smithi028.stdout:2.e 2 0 2 0 0 402 0 0 17 17 active+undersized+degraded 2021-05-12T09:10:17.320245+0000 51'101 53:128 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.030 INFO:tasks.workunit.client.0.smithi028.stdout:2.d 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321136+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.030 INFO:tasks.workunit.client.0.smithi028.stdout:2.c 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.322090+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.030 INFO:tasks.workunit.client.0.smithi028.stdout:2.b 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319604+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.031 INFO:tasks.workunit.client.0.smithi028.stdout:2.a 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319929+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.031 INFO:tasks.workunit.client.0.smithi028.stdout:2.9 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323963+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.031 INFO:tasks.workunit.client.0.smithi028.stdout:2.8 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323896+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.031 INFO:tasks.workunit.client.0.smithi028.stdout:2.7 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325446+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.032 INFO:tasks.workunit.client.0.smithi028.stdout:2.6 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.322282+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.032 INFO:tasks.workunit.client.0.smithi028.stdout:2.5 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318856+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.032 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 1 0 1 0 0 754 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321112+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.032 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:10:17.320695+0000 33'62 53:119 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:20.116394+0000 0'0 2021-05-12T09:09:20.116394+0000 0 2021-05-12T09:11:40.032 INFO:tasks.workunit.client.0.smithi028.stdout:2.3 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321973+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.033 INFO:tasks.workunit.client.0.smithi028.stdout:2.1 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321360+0000 33'100 53:126 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.033 INFO:tasks.workunit.client.0.smithi028.stdout:2.2 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323329+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.033 INFO:tasks.workunit.client.0.smithi028.stdout:2.4 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321691+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.033 INFO:tasks.workunit.client.0.smithi028.stdout:2.f 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.220557+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.034 INFO:tasks.workunit.client.0.smithi028.stdout:2.10 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321851+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.034 INFO:tasks.workunit.client.0.smithi028.stdout:2.11 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.324153+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.034 INFO:tasks.workunit.client.0.smithi028.stdout:2.12 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320164+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.034 INFO:tasks.workunit.client.0.smithi028.stdout:2.13 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318763+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.035 INFO:tasks.workunit.client.0.smithi028.stdout:2.14 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319539+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.035 INFO:tasks.workunit.client.0.smithi028.stdout:2.15 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318618+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.035 INFO:tasks.workunit.client.0.smithi028.stdout:2.16 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321253+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.035 INFO:tasks.workunit.client.0.smithi028.stdout:2.17 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.220474+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.035 INFO:tasks.workunit.client.0.smithi028.stdout:2.18 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319857+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.036 INFO:tasks.workunit.client.0.smithi028.stdout:2.19 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.321903+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.036 INFO:tasks.workunit.client.0.smithi028.stdout:2.1a 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.320526+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.036 INFO:tasks.workunit.client.0.smithi028.stdout:2.1b 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318500+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.037 INFO:tasks.workunit.client.0.smithi028.stdout:2.1c 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.319781+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.037 INFO:tasks.workunit.client.0.smithi028.stdout:2.1d 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.318542+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.037 INFO:tasks.workunit.client.0.smithi028.stdout:2.1e 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323197+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.037 INFO:tasks.workunit.client.0.smithi028.stdout:2.1f 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323951+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.037 INFO:tasks.workunit.client.0.smithi028.stdout:2.20 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323744+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.038 INFO:tasks.workunit.client.0.smithi028.stdout:2.21 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.323484+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.038 INFO:tasks.workunit.client.0.smithi028.stdout:2.22 2 0 2 0 0 755 0 0 16 16 active+undersized+degraded 2021-05-12T09:10:17.325216+0000 33'100 53:127 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.038 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:11:40.038 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:11:40.039 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:11:40.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2021-05-12T09:11:40.040 INFO:tasks.workunit.client.0.smithi028.stdout:wait for peering 2021-05-12T09:11:40.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2021-05-12T09:11:40.412 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:11:40.412 INFO:tasks.workunit.client.0.smithi028.stdout:2.3f 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069797+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.412 INFO:tasks.workunit.client.0.smithi028.stdout:2.3e 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947634+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.412 INFO:tasks.workunit.client.0.smithi028.stdout:2.3d 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.943781+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.413 INFO:tasks.workunit.client.0.smithi028.stdout:2.3c 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.068568+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.413 INFO:tasks.workunit.client.0.smithi028.stdout:2.3b 1 0 0 0 0 754 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.068952+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.413 INFO:tasks.workunit.client.0.smithi028.stdout:2.3a 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.070702+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.413 INFO:tasks.workunit.client.0.smithi028.stdout:2.39 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936132+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.414 INFO:tasks.workunit.client.0.smithi028.stdout:2.38 1 0 0 0 0 754 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.941851+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.414 INFO:tasks.workunit.client.0.smithi028.stdout:2.37 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947712+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.414 INFO:tasks.workunit.client.0.smithi028.stdout:2.36 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.070116+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.414 INFO:tasks.workunit.client.0.smithi028.stdout:2.35 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.937185+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.414 INFO:tasks.workunit.client.0.smithi028.stdout:2.34 1 0 0 0 0 754 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.944060+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.415 INFO:tasks.workunit.client.0.smithi028.stdout:2.33 1 0 0 0 0 754 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.944315+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.415 INFO:tasks.workunit.client.0.smithi028.stdout:2.32 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069267+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.415 INFO:tasks.workunit.client.0.smithi028.stdout:2.31 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947895+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.415 INFO:tasks.workunit.client.0.smithi028.stdout:2.30 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.948368+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.416 INFO:tasks.workunit.client.0.smithi028.stdout:2.2f 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.946339+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.416 INFO:tasks.workunit.client.0.smithi028.stdout:2.2e 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.944096+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.416 INFO:tasks.workunit.client.0.smithi028.stdout:2.2d 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936692+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.416 INFO:tasks.workunit.client.0.smithi028.stdout:2.2c 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.068390+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.416 INFO:tasks.workunit.client.0.smithi028.stdout:2.2b 1 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.070465+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.417 INFO:tasks.workunit.client.0.smithi028.stdout:2.2a 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.946058+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.417 INFO:tasks.workunit.client.0.smithi028.stdout:2.29 1 1 1 0 0 755 0 0 16 16 active+recovering+degraded+wait 2021-05-12T09:11:38.949993+0000 33'100 58:140 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.417 INFO:tasks.workunit.client.0.smithi028.stdout:2.28 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.946572+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.417 INFO:tasks.workunit.client.0.smithi028.stdout:2.27 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069865+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.418 INFO:tasks.workunit.client.0.smithi028.stdout:2.26 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.070120+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.418 INFO:tasks.workunit.client.0.smithi028.stdout:2.25 1 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.945787+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.418 INFO:tasks.workunit.client.0.smithi028.stdout:2.24 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.942993+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.418 INFO:tasks.workunit.client.0.smithi028.stdout:2.23 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069015+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.419 INFO:tasks.workunit.client.0.smithi028.stdout:2.e 2 1 1 0 0 402 0 0 14 14 active+recovering+degraded+wait 2021-05-12T09:11:39.071107+0000 51'101 58:140 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.419 INFO:tasks.workunit.client.0.smithi028.stdout:2.d 2 0 0 0 0 755 0 0 12 12 active+clean+wait 2021-05-12T09:11:39.068007+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.419 INFO:tasks.workunit.client.0.smithi028.stdout:2.c 2 0 0 0 0 755 0 0 14 14 active+clean+wait 2021-05-12T09:11:38.944292+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.419 INFO:tasks.workunit.client.0.smithi028.stdout:2.b 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.937835+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.420 INFO:tasks.workunit.client.0.smithi028.stdout:2.a 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.944815+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.420 INFO:tasks.workunit.client.0.smithi028.stdout:2.9 2 0 0 0 0 755 0 0 11 11 active+clean+wait 2021-05-12T09:11:39.069423+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.420 INFO:tasks.workunit.client.0.smithi028.stdout:2.8 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.069208+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.420 INFO:tasks.workunit.client.0.smithi028.stdout:2.7 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.945573+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.421 INFO:tasks.workunit.client.0.smithi028.stdout:2.6 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.942764+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.421 INFO:tasks.workunit.client.0.smithi028.stdout:2.5 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936981+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.421 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947208+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.421 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:11:38.933559+0000 33'62 58:132 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:20.116394+0000 0'0 2021-05-12T09:09:20.116394+0000 0 2021-05-12T09:11:40.422 INFO:tasks.workunit.client.0.smithi028.stdout:2.3 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.068275+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.422 INFO:tasks.workunit.client.0.smithi028.stdout:2.1 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.942229+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.422 INFO:tasks.workunit.client.0.smithi028.stdout:2.2 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.944546+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.422 INFO:tasks.workunit.client.0.smithi028.stdout:2.4 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.068632+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.422 INFO:tasks.workunit.client.0.smithi028.stdout:2.f 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936529+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.423 INFO:tasks.workunit.client.0.smithi028.stdout:2.10 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.942522+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.423 INFO:tasks.workunit.client.0.smithi028.stdout:2.11 2 1 1 0 0 755 0 0 16 16 active+recovering+degraded+wait 2021-05-12T09:11:38.953976+0000 33'100 58:140 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.423 INFO:tasks.workunit.client.0.smithi028.stdout:2.12 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947606+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.423 INFO:tasks.workunit.client.0.smithi028.stdout:2.13 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.939472+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.424 INFO:tasks.workunit.client.0.smithi028.stdout:2.14 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.937185+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.424 INFO:tasks.workunit.client.0.smithi028.stdout:2.15 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.846262+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.424 INFO:tasks.workunit.client.0.smithi028.stdout:2.16 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.943777+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.424 INFO:tasks.workunit.client.0.smithi028.stdout:2.17 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.934163+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.424 INFO:tasks.workunit.client.0.smithi028.stdout:2.18 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.067880+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.425 INFO:tasks.workunit.client.0.smithi028.stdout:2.19 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.067892+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.425 INFO:tasks.workunit.client.0.smithi028.stdout:2.1a 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.938620+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.425 INFO:tasks.workunit.client.0.smithi028.stdout:2.1b 2 1 1 0 0 755 0 0 16 16 active+recovering+degraded+wait 2021-05-12T09:11:38.845885+0000 33'100 58:140 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.425 INFO:tasks.workunit.client.0.smithi028.stdout:2.1c 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.938003+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.426 INFO:tasks.workunit.client.0.smithi028.stdout:2.1d 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936110+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.426 INFO:tasks.workunit.client.0.smithi028.stdout:2.1e 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.068276+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.426 INFO:tasks.workunit.client.0.smithi028.stdout:2.1f 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.943486+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.426 INFO:tasks.workunit.client.0.smithi028.stdout:2.20 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.944806+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.426 INFO:tasks.workunit.client.0.smithi028.stdout:2.21 2 0 0 0 0 755 0 0 15 15 active+clean+wait 2021-05-12T09:11:39.068508+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.427 INFO:tasks.workunit.client.0.smithi028.stdout:2.22 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069657+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:40.427 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:11:40.427 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:11:40.428 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:11:40.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2021-05-12T09:11:53.418 INFO:tasks.workunit.client.0.smithi028.stdout:killing divergent 2 2021-05-12T09:11:53.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 2' 2021-05-12T09:11:53.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2021-05-12T09:11:53.795 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:11:53.795 INFO:tasks.workunit.client.0.smithi028.stdout:2.3f 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069797+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.796 INFO:tasks.workunit.client.0.smithi028.stdout:2.3e 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947634+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.796 INFO:tasks.workunit.client.0.smithi028.stdout:2.3d 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.943781+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.796 INFO:tasks.workunit.client.0.smithi028.stdout:2.3c 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.068568+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.796 INFO:tasks.workunit.client.0.smithi028.stdout:2.3b 1 0 0 0 0 754 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.068952+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.797 INFO:tasks.workunit.client.0.smithi028.stdout:2.3a 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.070702+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.797 INFO:tasks.workunit.client.0.smithi028.stdout:2.39 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936132+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.797 INFO:tasks.workunit.client.0.smithi028.stdout:2.38 1 0 0 0 0 754 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.941851+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.798 INFO:tasks.workunit.client.0.smithi028.stdout:2.37 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947712+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.798 INFO:tasks.workunit.client.0.smithi028.stdout:2.36 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.070116+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.798 INFO:tasks.workunit.client.0.smithi028.stdout:2.35 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.937185+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.798 INFO:tasks.workunit.client.0.smithi028.stdout:2.34 1 0 0 0 0 754 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.944060+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.799 INFO:tasks.workunit.client.0.smithi028.stdout:2.33 1 0 0 0 0 754 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.944315+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.799 INFO:tasks.workunit.client.0.smithi028.stdout:2.32 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069267+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.799 INFO:tasks.workunit.client.0.smithi028.stdout:2.31 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947895+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.799 INFO:tasks.workunit.client.0.smithi028.stdout:2.30 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.948368+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.799 INFO:tasks.workunit.client.0.smithi028.stdout:2.2f 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.946339+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.800 INFO:tasks.workunit.client.0.smithi028.stdout:2.2e 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.944096+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.800 INFO:tasks.workunit.client.0.smithi028.stdout:2.2d 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936692+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.800 INFO:tasks.workunit.client.0.smithi028.stdout:2.2c 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.068390+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.800 INFO:tasks.workunit.client.0.smithi028.stdout:2.2b 1 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.070465+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.800 INFO:tasks.workunit.client.0.smithi028.stdout:2.2a 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.946058+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.801 INFO:tasks.workunit.client.0.smithi028.stdout:2.29 1 1 1 0 0 755 0 0 16 16 active+recovering+degraded+wait 2021-05-12T09:11:38.949993+0000 33'100 58:140 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.801 INFO:tasks.workunit.client.0.smithi028.stdout:2.28 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.946572+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.801 INFO:tasks.workunit.client.0.smithi028.stdout:2.27 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069865+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.802 INFO:tasks.workunit.client.0.smithi028.stdout:2.26 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.070120+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.802 INFO:tasks.workunit.client.0.smithi028.stdout:2.25 1 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.945787+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.802 INFO:tasks.workunit.client.0.smithi028.stdout:2.24 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.942993+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.803 INFO:tasks.workunit.client.0.smithi028.stdout:2.23 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069015+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.803 INFO:tasks.workunit.client.0.smithi028.stdout:2.e 2 1 1 0 0 402 0 0 14 14 active+recovering+degraded+wait 2021-05-12T09:11:39.071107+0000 51'101 58:140 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.803 INFO:tasks.workunit.client.0.smithi028.stdout:2.d 2 0 0 0 0 755 0 0 12 12 active+clean+wait 2021-05-12T09:11:39.068007+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.803 INFO:tasks.workunit.client.0.smithi028.stdout:2.c 2 0 0 0 0 755 0 0 14 14 active+clean+wait 2021-05-12T09:11:38.944292+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.803 INFO:tasks.workunit.client.0.smithi028.stdout:2.b 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.937835+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.804 INFO:tasks.workunit.client.0.smithi028.stdout:2.a 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.944815+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.804 INFO:tasks.workunit.client.0.smithi028.stdout:2.9 2 0 0 0 0 755 0 0 11 11 active+clean+wait 2021-05-12T09:11:39.069423+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.804 INFO:tasks.workunit.client.0.smithi028.stdout:2.8 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.069208+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.804 INFO:tasks.workunit.client.0.smithi028.stdout:2.7 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.945573+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.805 INFO:tasks.workunit.client.0.smithi028.stdout:2.6 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.942764+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.805 INFO:tasks.workunit.client.0.smithi028.stdout:2.5 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936981+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.806 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947208+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.806 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:11:38.933559+0000 33'62 58:132 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:20.116394+0000 0'0 2021-05-12T09:09:20.116394+0000 0 2021-05-12T09:11:53.806 INFO:tasks.workunit.client.0.smithi028.stdout:2.3 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.068275+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.806 INFO:tasks.workunit.client.0.smithi028.stdout:2.1 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.942229+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.807 INFO:tasks.workunit.client.0.smithi028.stdout:2.2 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.944546+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.807 INFO:tasks.workunit.client.0.smithi028.stdout:2.4 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.068632+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.807 INFO:tasks.workunit.client.0.smithi028.stdout:2.f 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936529+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.807 INFO:tasks.workunit.client.0.smithi028.stdout:2.10 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.942522+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.807 INFO:tasks.workunit.client.0.smithi028.stdout:2.11 2 1 1 0 0 755 0 0 16 16 active+recovering+degraded+wait 2021-05-12T09:11:38.953976+0000 33'100 58:140 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.808 INFO:tasks.workunit.client.0.smithi028.stdout:2.12 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947606+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.808 INFO:tasks.workunit.client.0.smithi028.stdout:2.13 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.939472+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.808 INFO:tasks.workunit.client.0.smithi028.stdout:2.14 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.937185+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.808 INFO:tasks.workunit.client.0.smithi028.stdout:2.15 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.846262+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.808 INFO:tasks.workunit.client.0.smithi028.stdout:2.16 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.943777+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.809 INFO:tasks.workunit.client.0.smithi028.stdout:2.17 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.934163+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.809 INFO:tasks.workunit.client.0.smithi028.stdout:2.18 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.067880+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.809 INFO:tasks.workunit.client.0.smithi028.stdout:2.19 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.067892+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.809 INFO:tasks.workunit.client.0.smithi028.stdout:2.1a 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.938620+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.810 INFO:tasks.workunit.client.0.smithi028.stdout:2.1b 2 1 1 0 0 755 0 0 16 16 active+recovering+degraded+wait 2021-05-12T09:11:38.845885+0000 33'100 58:140 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.810 INFO:tasks.workunit.client.0.smithi028.stdout:2.1c 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.938003+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.810 INFO:tasks.workunit.client.0.smithi028.stdout:2.1d 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936110+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.810 INFO:tasks.workunit.client.0.smithi028.stdout:2.1e 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.068276+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.810 INFO:tasks.workunit.client.0.smithi028.stdout:2.1f 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.943486+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.811 INFO:tasks.workunit.client.0.smithi028.stdout:2.20 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.944806+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.811 INFO:tasks.workunit.client.0.smithi028.stdout:2.21 2 0 0 0 0 755 0 0 15 15 active+clean+wait 2021-05-12T09:11:39.068508+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.811 INFO:tasks.workunit.client.0.smithi028.stdout:2.22 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069657+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:53.811 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:11:53.812 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:11:53.813 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:11:53.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.2 2021-05-12T09:11:53.814 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:11:53.814 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:11:53.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:11:53.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:11:53.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:11:53.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:11:53.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.29719.out 2021-05-12T09:11:53.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 2 --op export-remove --pgid 2.0 --file td/divergent-priors/exp.29719.out 2021-05-12T09:11:53.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T09:11:53.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T09:11:53.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=2 2021-05-12T09:11:53.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T09:11:53.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2021-05-12T09:11:53.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op export-remove --pgid 2.0 --file td/divergent-priors/exp.29719.out 2021-05-12T09:11:54.761 INFO:tasks.workunit.client.0.smithi028.stderr:Exporting 2.0 info 2.0( v 33'100 (33'84,33'100] local-lis/les=56/58 n=1 ec=25/25 lis/c=56/56 les/c/f=58/58/0 sis=56 pruub=14.396658897s) 2021-05-12T09:11:54.764 INFO:tasks.workunit.client.0.smithi028.stderr:Read #2:023dd3d2:::existing_12:head# 2021-05-12T09:11:54.793 INFO:tasks.workunit.client.0.smithi028.stderr:Export successful 2021-05-12T09:11:54.794 INFO:tasks.workunit.client.0.smithi028.stdout: marking collection for removal 2021-05-12T09:11:54.795 INFO:tasks.workunit.client.0.smithi028.stdout:setting '_remove' omap key 2021-05-12T09:11:54.795 INFO:tasks.workunit.client.0.smithi028.stdout:finish_remove_pgs 2.0_head removing 2.0 2021-05-12T09:11:54.926 INFO:tasks.workunit.client.0.smithi028.stderr:Remove successful 2021-05-12T09:11:55.310 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 2 --op import --file td/divergent-priors/exp.29719.out 2021-05-12T09:11:55.310 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T09:11:55.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T09:11:55.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=2 2021-05-12T09:11:55.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T09:11:55.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2021-05-12T09:11:55.312 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op import --file td/divergent-priors/exp.29719.out 2021-05-12T09:11:56.003 INFO:tasks.workunit.client.0.smithi028.stdout:get_pg_num_history pg_num_history pg_num_history(e58 pg_nums {1={16=1},2={25=1,35=64}} deleted_pools ) 2021-05-12T09:11:56.003 INFO:tasks.workunit.client.0.smithi028.stdout:Importing pgid 2.0 2021-05-12T09:11:56.004 INFO:tasks.workunit.client.0.smithi028.stdout:Write #2:023dd3d2:::existing_12:head# 2021-05-12T09:11:56.004 INFO:tasks.workunit.client.0.smithi028.stdout:snapset 0=[]:{} 2021-05-12T09:11:56.061 INFO:tasks.workunit.client.0.smithi028.stdout:write_pg epoch 58 info 2.0( v 33'100 (33'84,33'100] local-lis/les=56/58 n=1 ec=25/25 lis/c=56/56 les/c/f=58/58/0 sis=56 pruub=14.396658897s) 2021-05-12T09:11:56.087 INFO:tasks.workunit.client.0.smithi028.stdout:Import successful 2021-05-12T09:11:56.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 2' 2021-05-12T09:11:56.604 INFO:tasks.workunit.client.0.smithi028.stdout:reviving divergent 2 2021-05-12T09:11:56.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2021-05-12T09:11:56.980 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:11:56.980 INFO:tasks.workunit.client.0.smithi028.stdout:2.3f 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069797+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.981 INFO:tasks.workunit.client.0.smithi028.stdout:2.3e 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947634+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.981 INFO:tasks.workunit.client.0.smithi028.stdout:2.3d 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.943781+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.981 INFO:tasks.workunit.client.0.smithi028.stdout:2.3c 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.068568+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.982 INFO:tasks.workunit.client.0.smithi028.stdout:2.3b 1 0 0 0 0 754 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.068952+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.982 INFO:tasks.workunit.client.0.smithi028.stdout:2.3a 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.070702+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.983 INFO:tasks.workunit.client.0.smithi028.stdout:2.39 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936132+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.983 INFO:tasks.workunit.client.0.smithi028.stdout:2.38 1 0 0 0 0 754 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.941851+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.983 INFO:tasks.workunit.client.0.smithi028.stdout:2.37 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947712+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.984 INFO:tasks.workunit.client.0.smithi028.stdout:2.36 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.070116+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.984 INFO:tasks.workunit.client.0.smithi028.stdout:2.35 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.937185+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.985 INFO:tasks.workunit.client.0.smithi028.stdout:2.34 1 0 0 0 0 754 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.944060+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.985 INFO:tasks.workunit.client.0.smithi028.stdout:2.33 1 0 0 0 0 754 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.944315+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.986 INFO:tasks.workunit.client.0.smithi028.stdout:2.32 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069267+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.986 INFO:tasks.workunit.client.0.smithi028.stdout:2.31 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947895+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.986 INFO:tasks.workunit.client.0.smithi028.stdout:2.30 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.948368+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.987 INFO:tasks.workunit.client.0.smithi028.stdout:2.2f 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.946339+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.987 INFO:tasks.workunit.client.0.smithi028.stdout:2.2e 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.944096+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.988 INFO:tasks.workunit.client.0.smithi028.stdout:2.2d 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936692+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.988 INFO:tasks.workunit.client.0.smithi028.stdout:2.2c 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.068390+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.989 INFO:tasks.workunit.client.0.smithi028.stdout:2.2b 1 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.070465+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.989 INFO:tasks.workunit.client.0.smithi028.stdout:2.2a 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.946058+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.989 INFO:tasks.workunit.client.0.smithi028.stdout:2.29 1 1 1 0 0 755 0 0 16 16 active+recovering+degraded+wait 2021-05-12T09:11:38.949993+0000 33'100 58:140 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.990 INFO:tasks.workunit.client.0.smithi028.stdout:2.28 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.946572+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.990 INFO:tasks.workunit.client.0.smithi028.stdout:2.27 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069865+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.991 INFO:tasks.workunit.client.0.smithi028.stdout:2.26 1 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.070120+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.991 INFO:tasks.workunit.client.0.smithi028.stdout:2.25 1 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.945787+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.992 INFO:tasks.workunit.client.0.smithi028.stdout:2.24 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.942993+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.992 INFO:tasks.workunit.client.0.smithi028.stdout:2.23 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069015+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.992 INFO:tasks.workunit.client.0.smithi028.stdout:2.e 2 1 1 0 0 402 0 0 14 14 active+recovering+degraded+wait 2021-05-12T09:11:39.071107+0000 51'101 58:140 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.993 INFO:tasks.workunit.client.0.smithi028.stdout:2.d 2 0 0 0 0 755 0 0 12 12 active+clean+wait 2021-05-12T09:11:39.068007+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.993 INFO:tasks.workunit.client.0.smithi028.stdout:2.c 2 0 0 0 0 755 0 0 14 14 active+clean+wait 2021-05-12T09:11:38.944292+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.994 INFO:tasks.workunit.client.0.smithi028.stdout:2.b 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.937835+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.994 INFO:tasks.workunit.client.0.smithi028.stdout:2.a 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.944815+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.995 INFO:tasks.workunit.client.0.smithi028.stdout:2.9 2 0 0 0 0 755 0 0 11 11 active+clean+wait 2021-05-12T09:11:39.069423+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.995 INFO:tasks.workunit.client.0.smithi028.stdout:2.8 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.069208+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.995 INFO:tasks.workunit.client.0.smithi028.stdout:2.7 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.945573+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.996 INFO:tasks.workunit.client.0.smithi028.stdout:2.6 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.942764+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.996 INFO:tasks.workunit.client.0.smithi028.stdout:2.5 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936981+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.997 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 1 0 0 0 0 754 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947208+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.997 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:11:38.933559+0000 33'62 58:132 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:20.116394+0000 0'0 2021-05-12T09:09:20.116394+0000 0 2021-05-12T09:11:56.997 INFO:tasks.workunit.client.0.smithi028.stdout:2.3 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.068275+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.998 INFO:tasks.workunit.client.0.smithi028.stdout:2.1 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.942229+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.998 INFO:tasks.workunit.client.0.smithi028.stdout:2.2 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.944546+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.999 INFO:tasks.workunit.client.0.smithi028.stdout:2.4 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.068632+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:56.999 INFO:tasks.workunit.client.0.smithi028.stdout:2.f 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936529+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.000 INFO:tasks.workunit.client.0.smithi028.stdout:2.10 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.942522+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.000 INFO:tasks.workunit.client.0.smithi028.stdout:2.11 2 1 1 0 0 755 0 0 16 16 active+recovering+degraded+wait 2021-05-12T09:11:38.953976+0000 33'100 58:140 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.001 INFO:tasks.workunit.client.0.smithi028.stdout:2.12 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.947606+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.001 INFO:tasks.workunit.client.0.smithi028.stdout:2.13 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.939472+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.001 INFO:tasks.workunit.client.0.smithi028.stdout:2.14 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.937185+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.002 INFO:tasks.workunit.client.0.smithi028.stdout:2.15 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.846262+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.002 INFO:tasks.workunit.client.0.smithi028.stdout:2.16 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.943777+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.003 INFO:tasks.workunit.client.0.smithi028.stdout:2.17 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.934163+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.003 INFO:tasks.workunit.client.0.smithi028.stdout:2.18 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.067880+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.004 INFO:tasks.workunit.client.0.smithi028.stdout:2.19 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.067892+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.004 INFO:tasks.workunit.client.0.smithi028.stdout:2.1a 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.938620+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.004 INFO:tasks.workunit.client.0.smithi028.stdout:2.1b 2 1 1 0 0 755 0 0 16 16 active+recovering+degraded+wait 2021-05-12T09:11:38.845885+0000 33'100 58:140 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.005 INFO:tasks.workunit.client.0.smithi028.stdout:2.1c 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.938003+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.005 INFO:tasks.workunit.client.0.smithi028.stdout:2.1d 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.936110+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.006 INFO:tasks.workunit.client.0.smithi028.stdout:2.1e 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:39.068276+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.006 INFO:tasks.workunit.client.0.smithi028.stdout:2.1f 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:38.943486+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.007 INFO:tasks.workunit.client.0.smithi028.stdout:2.20 2 0 0 0 0 755 0 0 10 10 active+clean+wait 2021-05-12T09:11:38.944806+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.007 INFO:tasks.workunit.client.0.smithi028.stdout:2.21 2 0 0 0 0 755 0 0 15 15 active+clean+wait 2021-05-12T09:11:39.068508+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.007 INFO:tasks.workunit.client.0.smithi028.stdout:2.22 2 0 0 0 0 755 0 0 16 16 active+clean+wait 2021-05-12T09:11:39.069657+0000 33'100 58:141 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:11:57.008 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:11:57.008 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:11:57.012 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:11:57.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 2 2021-05-12T09:11:57.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:11:57.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:11:57.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T09:11:57.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:11:57.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T09:11:57.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:11:57.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:11:57.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:11:57.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:11:57.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T09:11:57.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T09:11:57.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:11:57.015 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:11:57.015 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:11:57.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:11:57.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:11:57.016 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:11:57.016 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:11:57.016 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:11:57.016 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:11:57.016 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:11:57.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:11:57.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:11:57.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:11:57.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:11:57.018 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:11:57.018 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:11:57.018 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:11:57.018 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:11:57.018 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:11:57.019 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T09:11:57.019 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T09:11:57.019 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:11:57.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:11:57.021 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T09:11:57.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T09:11:57.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:11:57.023 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:11:57.023 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:11:57.024 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:11:57.023+0000 7ffbd8e3a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:11:57.025 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:11:57.025+0000 7ffbd8e3a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:11:57.026 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:11:57.026+0000 7ffbd8e3a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:11:57.382 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T09:11:57.382 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:11:57.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:11:57.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:11:57.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:11:57.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:11:57.385 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:11:57.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:11:57.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:11:57.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:11:57.544 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:11:57.544+0000 7ffbd8e3a0c0 -1 Falling back to public interface 2021-05-12T09:11:57.779 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 56 up_thru 56 down_at 39 last_clean_interval [15,36) [v2:127.0.0.1:6818/66289,v1:127.0.0.1:6819/66289] [v2:127.0.0.1:6820/66289,v1:127.0.0.1:6821/66289] exists,up cabac835-d2f0-45a7-ba67-30ece6d00d83 2021-05-12T09:11:57.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:11:57.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:11:57.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:11:57.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 2 2021-05-12T09:11:57.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:11:57.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:11:57.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:11:57.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:11:57.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:11:57.783 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:11:57.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:11:57.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:11:57.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:11:58.200 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 56 up_thru 56 down_at 39 last_clean_interval [15,36) [v2:127.0.0.1:6818/66289,v1:127.0.0.1:6819/66289] [v2:127.0.0.1:6820/66289,v1:127.0.0.1:6821/66289] exists,up cabac835-d2f0-45a7-ba67-30ece6d00d83 2021-05-12T09:11:58.200 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:11:58.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:11:58.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:11:58.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2021-05-12T09:11:59.917 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:11:59.915+0000 7ffbd8e3a0c0 -1 osd.2 58 log_to_monitors {default=true} 2021-05-12T09:12:18.202 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.2 2021-05-12T09:12:18.203 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T09:12:18.203 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T09:12:18.203 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:12:18.204 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:12:18.204 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:12:18.204 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.2.asok 2021-05-12T09:12:18.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2021-05-12T09:12:18.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.2.asok dump_ops_in_flight 2021-05-12T09:12:18.311 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:12:18.311 INFO:tasks.workunit.client.0.smithi028.stdout: "ops": [], 2021-05-12T09:12:18.311 INFO:tasks.workunit.client.0.smithi028.stdout: "num_ops": 0 2021-05-12T09:12:18.312 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:12:18.320 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2021-05-12T09:12:18.321 INFO:tasks.workunit.client.0.smithi028.stdout:allowing recovery 2021-05-12T09:12:18.321 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2021-05-12T09:12:18.695 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:12:18.696 INFO:tasks.workunit.client.0.smithi028.stdout:2.3f 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:11.107770+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.696 INFO:tasks.workunit.client.0.smithi028.stdout:2.3e 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:08.937418+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.696 INFO:tasks.workunit.client.0.smithi028.stdout:2.3d 1 0 0 0 0 754 0 0 16 16 activating 2021-05-12T09:12:14.710926+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.696 INFO:tasks.workunit.client.0.smithi028.stdout:2.3c 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:13.080617+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.697 INFO:tasks.workunit.client.0.smithi028.stdout:2.3b 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:11.108253+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.697 INFO:tasks.workunit.client.0.smithi028.stdout:2.3a 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:08.938621+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.697 INFO:tasks.workunit.client.0.smithi028.stdout:2.39 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:01.562981+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.697 INFO:tasks.workunit.client.0.smithi028.stdout:2.38 1 0 0 0 0 754 0 0 10 10 active+clean 2021-05-12T09:12:01.563362+0000 33'100 71:162 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.698 INFO:tasks.workunit.client.0.smithi028.stdout:2.37 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:11.106977+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.698 INFO:tasks.workunit.client.0.smithi028.stdout:2.36 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:08.937860+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.698 INFO:tasks.workunit.client.0.smithi028.stdout:2.35 1 0 0 0 0 754 0 0 16 16 activating 2021-05-12T09:12:14.710385+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.699 INFO:tasks.workunit.client.0.smithi028.stdout:2.34 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:13.081230+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.699 INFO:tasks.workunit.client.0.smithi028.stdout:2.33 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:11.109905+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.699 INFO:tasks.workunit.client.0.smithi028.stdout:2.32 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:08.938650+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.700 INFO:tasks.workunit.client.0.smithi028.stdout:2.31 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:01.557310+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.700 INFO:tasks.workunit.client.0.smithi028.stdout:2.30 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:01.562940+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.700 INFO:tasks.workunit.client.0.smithi028.stdout:2.2f 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:11.109678+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.700 INFO:tasks.workunit.client.0.smithi028.stdout:2.2e 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:08.933877+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.701 INFO:tasks.workunit.client.0.smithi028.stdout:2.2d 1 0 0 0 0 754 0 0 16 16 activating 2021-05-12T09:12:14.710295+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.701 INFO:tasks.workunit.client.0.smithi028.stdout:2.2c 1 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:12.989031+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.701 INFO:tasks.workunit.client.0.smithi028.stdout:2.2b 1 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.108081+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.701 INFO:tasks.workunit.client.0.smithi028.stdout:2.2a 1 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:08.937892+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.702 INFO:tasks.workunit.client.0.smithi028.stdout:2.29 1 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:05.080141+0000 33'100 71:164 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.702 INFO:tasks.workunit.client.0.smithi028.stdout:2.28 1 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:01.561809+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.702 INFO:tasks.workunit.client.0.smithi028.stdout:2.27 1 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.108686+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.702 INFO:tasks.workunit.client.0.smithi028.stdout:2.26 1 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:08.939003+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.703 INFO:tasks.workunit.client.0.smithi028.stdout:2.25 1 0 0 0 0 755 0 0 16 16 activating 2021-05-12T09:12:14.710411+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.703 INFO:tasks.workunit.client.0.smithi028.stdout:2.24 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:12.988581+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.703 INFO:tasks.workunit.client.0.smithi028.stdout:2.23 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.107197+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.704 INFO:tasks.workunit.client.0.smithi028.stdout:2.e 2 0 0 0 0 402 0 0 17 17 active+clean 2021-05-12T09:12:08.938104+0000 51'101 71:141 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.704 INFO:tasks.workunit.client.0.smithi028.stdout:2.d 2 0 0 0 0 755 0 0 16 16 activating 2021-05-12T09:12:14.711003+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.704 INFO:tasks.workunit.client.0.smithi028.stdout:2.c 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:13.081481+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.704 INFO:tasks.workunit.client.0.smithi028.stdout:2.b 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T09:12:12.646425+0000 33'100 71:150 [1,2,0] 1 [1,2,0] 1 33'100 2021-05-12T09:12:12.646354+0000 33'100 2021-05-12T09:12:12.646354+0000 0 2021-05-12T09:12:18.704 INFO:tasks.workunit.client.0.smithi028.stdout:2.a 3 0 0 0 0 1449 0 0 16 16 active+clean 2021-05-12T09:12:10.670798+0000 33'100 71:145 [0,1,2] 0 [0,1,2] 0 33'100 2021-05-12T09:12:10.670745+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.705 INFO:tasks.workunit.client.0.smithi028.stdout:2.9 2 0 0 0 0 755 0 0 11 11 active+clean 2021-05-12T09:12:01.616769+0000 33'100 71:162 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.705 INFO:tasks.workunit.client.0.smithi028.stdout:2.8 2 0 0 0 0 755 0 0 10 10 active+clean 2021-05-12T09:12:01.565275+0000 33'100 71:162 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.705 INFO:tasks.workunit.client.0.smithi028.stdout:2.7 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.109377+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.705 INFO:tasks.workunit.client.0.smithi028.stdout:2.6 3 0 0 0 0 1238 0 0 17 17 active+clean 2021-05-12T09:12:08.899850+0000 58'101 71:159 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.706 INFO:tasks.workunit.client.0.smithi028.stdout:2.5 2 0 0 0 0 755 0 0 16 16 activating 2021-05-12T09:12:14.711034+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.706 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:01.562114+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.706 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:12:01.550589+0000 33'62 71:156 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:20.116394+0000 0'0 2021-05-12T09:09:20.116394+0000 0 2021-05-12T09:12:18.706 INFO:tasks.workunit.client.0.smithi028.stdout:2.3 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.107487+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.707 INFO:tasks.workunit.client.0.smithi028.stdout:2.1 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:01.563727+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.707 INFO:tasks.workunit.client.0.smithi028.stdout:2.2 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:08.937644+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.707 INFO:tasks.workunit.client.0.smithi028.stdout:2.4 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:13.080651+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.707 INFO:tasks.workunit.client.0.smithi028.stdout:2.f 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T09:12:14.542669+0000 33'100 71:150 [1,2,0] 1 [1,2,0] 1 33'100 2021-05-12T09:12:14.542600+0000 33'100 2021-05-12T09:12:14.542600+0000 0 2021-05-12T09:12:18.707 INFO:tasks.workunit.client.0.smithi028.stdout:2.10 2 0 0 0 0 755 0 0 10 10 active+clean 2021-05-12T09:12:01.564072+0000 33'100 71:162 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.708 INFO:tasks.workunit.client.0.smithi028.stdout:2.11 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:10.285937+0000 33'100 71:164 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.708 INFO:tasks.workunit.client.0.smithi028.stdout:2.12 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:08.938762+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.708 INFO:tasks.workunit.client.0.smithi028.stdout:2.13 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T09:12:11.357211+0000 33'100 71:150 [1,2,0] 1 [1,2,0] 1 33'100 2021-05-12T09:12:11.357145+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.708 INFO:tasks.workunit.client.0.smithi028.stdout:2.14 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:12.988929+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.709 INFO:tasks.workunit.client.0.smithi028.stdout:2.15 2 0 0 0 0 755 0 0 16 16 activating 2021-05-12T09:12:14.710411+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.709 INFO:tasks.workunit.client.0.smithi028.stdout:2.16 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:08.900088+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.709 INFO:tasks.workunit.client.0.smithi028.stdout:2.17 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.022676+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.709 INFO:tasks.workunit.client.0.smithi028.stdout:2.18 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:01.563800+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.710 INFO:tasks.workunit.client.0.smithi028.stdout:2.19 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:01.562990+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.710 INFO:tasks.workunit.client.0.smithi028.stdout:2.1a 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T09:12:11.569449+0000 33'100 71:145 [0,1,2] 0 [0,1,2] 0 33'100 2021-05-12T09:12:11.569384+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.710 INFO:tasks.workunit.client.0.smithi028.stdout:2.1b 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.109003+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.710 INFO:tasks.workunit.client.0.smithi028.stdout:2.1c 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:12.989002+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.710 INFO:tasks.workunit.client.0.smithi028.stdout:2.1d 2 0 0 0 0 755 0 0 16 16 activating 2021-05-12T09:12:14.710385+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.711 INFO:tasks.workunit.client.0.smithi028.stdout:2.1e 3 0 0 0 0 1449 0 0 16 16 active+clean 2021-05-12T09:12:13.496715+0000 33'100 71:145 [0,1,2] 0 [0,1,2] 0 33'100 2021-05-12T09:12:13.496647+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.711 INFO:tasks.workunit.client.0.smithi028.stdout:2.1f 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.109658+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.711 INFO:tasks.workunit.client.0.smithi028.stdout:2.20 2 0 0 0 0 755 0 0 10 10 active+clean 2021-05-12T09:12:01.564662+0000 33'100 71:162 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.711 INFO:tasks.workunit.client.0.smithi028.stdout:2.21 2 0 0 0 0 755 0 0 15 15 active+clean 2021-05-12T09:12:01.564567+0000 33'100 71:162 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.712 INFO:tasks.workunit.client.0.smithi028.stdout:2.22 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:08.938393+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:18.712 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:12:18.712 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:12:18.713 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:12:18.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-05-12T09:12:18.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2021-05-12T09:12:18.837 INFO:tasks.workunit.client.0.smithi028.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T09:12:18.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-05-12T09:12:18.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2021-05-12T09:12:18.977 INFO:tasks.workunit.client.0.smithi028.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T09:12:18.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-05-12T09:12:18.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2021-05-12T09:12:19.118 INFO:tasks.workunit.client.0.smithi028.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T09:12:19.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2021-05-12T09:12:19.127 INFO:tasks.workunit.client.0.smithi028.stdout:reading divergent objects 2021-05-12T09:12:19.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2021-05-12T09:12:19.497 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:12:19.497 INFO:tasks.workunit.client.0.smithi028.stdout:2.3f 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:11.107770+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.498 INFO:tasks.workunit.client.0.smithi028.stdout:2.3e 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:08.937418+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.498 INFO:tasks.workunit.client.0.smithi028.stdout:2.3d 1 0 0 0 0 754 0 0 16 16 activating 2021-05-12T09:12:14.710926+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.498 INFO:tasks.workunit.client.0.smithi028.stdout:2.3c 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:13.080617+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.499 INFO:tasks.workunit.client.0.smithi028.stdout:2.3b 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:11.108253+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.499 INFO:tasks.workunit.client.0.smithi028.stdout:2.3a 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:08.938621+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.499 INFO:tasks.workunit.client.0.smithi028.stdout:2.39 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:01.562981+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.499 INFO:tasks.workunit.client.0.smithi028.stdout:2.38 1 0 0 0 0 754 0 0 10 10 active+clean 2021-05-12T09:12:01.563362+0000 33'100 71:162 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.499 INFO:tasks.workunit.client.0.smithi028.stdout:2.37 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:11.106977+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.500 INFO:tasks.workunit.client.0.smithi028.stdout:2.36 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:08.937860+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.500 INFO:tasks.workunit.client.0.smithi028.stdout:2.35 1 0 0 0 0 754 0 0 16 16 activating 2021-05-12T09:12:14.710385+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.500 INFO:tasks.workunit.client.0.smithi028.stdout:2.34 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:13.081230+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.500 INFO:tasks.workunit.client.0.smithi028.stdout:2.33 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:11.109905+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.501 INFO:tasks.workunit.client.0.smithi028.stdout:2.32 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:08.938650+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.501 INFO:tasks.workunit.client.0.smithi028.stdout:2.31 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:01.557310+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.501 INFO:tasks.workunit.client.0.smithi028.stdout:2.30 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:01.562940+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.501 INFO:tasks.workunit.client.0.smithi028.stdout:2.2f 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:11.109678+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.502 INFO:tasks.workunit.client.0.smithi028.stdout:2.2e 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:08.933877+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.502 INFO:tasks.workunit.client.0.smithi028.stdout:2.2d 1 0 0 0 0 754 0 0 16 16 activating 2021-05-12T09:12:14.710295+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.502 INFO:tasks.workunit.client.0.smithi028.stdout:2.2c 1 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:12.989031+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.503 INFO:tasks.workunit.client.0.smithi028.stdout:2.2b 1 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.108081+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.504 INFO:tasks.workunit.client.0.smithi028.stdout:2.2a 1 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:08.937892+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.505 INFO:tasks.workunit.client.0.smithi028.stdout:2.29 1 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:05.080141+0000 33'100 71:164 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.505 INFO:tasks.workunit.client.0.smithi028.stdout:2.28 1 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:01.561809+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.505 INFO:tasks.workunit.client.0.smithi028.stdout:2.27 1 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.108686+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.505 INFO:tasks.workunit.client.0.smithi028.stdout:2.26 1 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:08.939003+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.506 INFO:tasks.workunit.client.0.smithi028.stdout:2.25 1 0 0 0 0 755 0 0 16 16 activating 2021-05-12T09:12:14.710411+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.506 INFO:tasks.workunit.client.0.smithi028.stdout:2.24 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:12.988581+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.506 INFO:tasks.workunit.client.0.smithi028.stdout:2.23 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.107197+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.506 INFO:tasks.workunit.client.0.smithi028.stdout:2.e 2 0 0 0 0 402 0 0 17 17 active+clean 2021-05-12T09:12:08.938104+0000 51'101 71:141 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.507 INFO:tasks.workunit.client.0.smithi028.stdout:2.d 2 0 0 0 0 755 0 0 16 16 activating 2021-05-12T09:12:14.711003+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.507 INFO:tasks.workunit.client.0.smithi028.stdout:2.c 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:13.081481+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.507 INFO:tasks.workunit.client.0.smithi028.stdout:2.b 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T09:12:12.646425+0000 33'100 71:150 [1,2,0] 1 [1,2,0] 1 33'100 2021-05-12T09:12:12.646354+0000 33'100 2021-05-12T09:12:12.646354+0000 0 2021-05-12T09:12:19.507 INFO:tasks.workunit.client.0.smithi028.stdout:2.a 3 0 0 0 0 1449 0 0 16 16 active+clean 2021-05-12T09:12:10.670798+0000 33'100 71:145 [0,1,2] 0 [0,1,2] 0 33'100 2021-05-12T09:12:10.670745+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.508 INFO:tasks.workunit.client.0.smithi028.stdout:2.9 2 0 0 0 0 755 0 0 11 11 active+clean 2021-05-12T09:12:01.616769+0000 33'100 71:162 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.508 INFO:tasks.workunit.client.0.smithi028.stdout:2.8 2 0 0 0 0 755 0 0 10 10 active+clean 2021-05-12T09:12:01.565275+0000 33'100 71:162 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.508 INFO:tasks.workunit.client.0.smithi028.stdout:2.7 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.109377+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.508 INFO:tasks.workunit.client.0.smithi028.stdout:2.6 3 0 0 0 0 1238 0 0 17 17 active+clean 2021-05-12T09:12:08.899850+0000 58'101 71:159 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.508 INFO:tasks.workunit.client.0.smithi028.stdout:2.5 2 0 0 0 0 755 0 0 16 16 activating 2021-05-12T09:12:14.711034+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.509 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 1 0 0 0 0 754 0 0 16 16 active+clean 2021-05-12T09:12:01.562114+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.509 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:12:01.550589+0000 33'62 71:156 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:20.116394+0000 0'0 2021-05-12T09:09:20.116394+0000 0 2021-05-12T09:12:19.509 INFO:tasks.workunit.client.0.smithi028.stdout:2.3 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.107487+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.510 INFO:tasks.workunit.client.0.smithi028.stdout:2.1 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:01.563727+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.510 INFO:tasks.workunit.client.0.smithi028.stdout:2.2 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:08.937644+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.511 INFO:tasks.workunit.client.0.smithi028.stdout:2.4 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:13.080651+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.511 INFO:tasks.workunit.client.0.smithi028.stdout:2.f 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T09:12:14.542669+0000 33'100 71:150 [1,2,0] 1 [1,2,0] 1 33'100 2021-05-12T09:12:14.542600+0000 33'100 2021-05-12T09:12:14.542600+0000 0 2021-05-12T09:12:19.511 INFO:tasks.workunit.client.0.smithi028.stdout:2.10 2 0 0 0 0 755 0 0 10 10 active+clean 2021-05-12T09:12:01.564072+0000 33'100 71:162 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.511 INFO:tasks.workunit.client.0.smithi028.stdout:2.11 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:10.285937+0000 33'100 71:164 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.512 INFO:tasks.workunit.client.0.smithi028.stdout:2.12 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:08.938762+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.512 INFO:tasks.workunit.client.0.smithi028.stdout:2.13 0 0 0 0 0 0 0 0 16 16 active+clean 2021-05-12T09:12:11.357211+0000 33'100 71:150 [1,2,0] 1 [1,2,0] 1 33'100 2021-05-12T09:12:11.357145+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.512 INFO:tasks.workunit.client.0.smithi028.stdout:2.14 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:12.988929+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.514 INFO:tasks.workunit.client.0.smithi028.stdout:2.15 2 0 0 0 0 755 0 0 16 16 activating 2021-05-12T09:12:14.710411+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.514 INFO:tasks.workunit.client.0.smithi028.stdout:2.16 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:08.900088+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.515 INFO:tasks.workunit.client.0.smithi028.stdout:2.17 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.022676+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.515 INFO:tasks.workunit.client.0.smithi028.stdout:2.18 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:01.563800+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.515 INFO:tasks.workunit.client.0.smithi028.stdout:2.19 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:01.562990+0000 33'100 71:161 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.515 INFO:tasks.workunit.client.0.smithi028.stdout:2.1a 1 0 0 0 0 483 0 0 16 16 active+clean 2021-05-12T09:12:11.569449+0000 33'100 71:145 [0,1,2] 0 [0,1,2] 0 33'100 2021-05-12T09:12:11.569384+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.516 INFO:tasks.workunit.client.0.smithi028.stdout:2.1b 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.109003+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.516 INFO:tasks.workunit.client.0.smithi028.stdout:2.1c 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:12.989002+0000 33'100 71:143 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.516 INFO:tasks.workunit.client.0.smithi028.stdout:2.1d 2 0 0 0 0 755 0 0 16 16 activating 2021-05-12T09:12:14.710385+0000 33'100 72:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.516 INFO:tasks.workunit.client.0.smithi028.stdout:2.1e 3 0 0 0 0 1449 0 0 16 16 active+clean 2021-05-12T09:12:13.496715+0000 33'100 71:145 [0,1,2] 0 [0,1,2] 0 33'100 2021-05-12T09:12:13.496647+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.517 INFO:tasks.workunit.client.0.smithi028.stdout:2.1f 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:11.109658+0000 33'100 71:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.517 INFO:tasks.workunit.client.0.smithi028.stdout:2.20 2 0 0 0 0 755 0 0 10 10 active+clean 2021-05-12T09:12:01.564662+0000 33'100 71:162 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.517 INFO:tasks.workunit.client.0.smithi028.stdout:2.21 2 0 0 0 0 755 0 0 15 15 active+clean 2021-05-12T09:12:01.564567+0000 33'100 71:162 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.518 INFO:tasks.workunit.client.0.smithi028.stdout:2.22 2 0 0 0 0 755 0 0 16 16 active+clean 2021-05-12T09:12:08.938393+0000 33'100 71:140 [0,1,2] 0 [0,1,2] 0 0'0 2021-05-12T09:09:23.109566+0000 0'0 2021-05-12T09:09:23.109566+0000 0 2021-05-12T09:12:19.518 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:12:19.519 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:12:19.520 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:12:19.520 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2021-05-12T09:12:19.520 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2021-05-12T09:12:19.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:12:19.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2021-05-12T09:12:19.566 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:12:19.567 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2021-05-12T09:12:19.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:12:19.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2021-05-12T09:12:19.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:12:19.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2021-05-12T09:12:19.675 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2021-05-12T09:12:19.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T09:12:19.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2021-05-12T09:12:19.705 INFO:tasks.workunit.client.0.smithi028.stderr:error getting test/newobject_1: (2) No such file or directory 2021-05-12T09:12:19.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-05-12T09:12:19.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2021-05-12T09:12:19.736 INFO:tasks.workunit.client.0.smithi028.stderr:error getting test/newobject_2: (2) No such file or directory 2021-05-12T09:12:19.738 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2021-05-12T09:12:19.739 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-05-12T09:12:19.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T09:12:19.758 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 20 _merge_object_divergent_entries: merging hoid 2:d858c035:::existing_1:head entries: 33'101 (33'1) modify 2:d858c035:::existing_1:head by client.4538.0:1 2021-05-12T09:09:54.139229+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:12:19.758 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 20 _merge_object_divergent_entries: keeping 33'101 (33'1) modify 2:d858c035:::existing_1:head by client.4538.0:1 2021-05-12T09:09:54.139229+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:12:19.759 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T09:12:19.759 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:d858c035:::existing_1:head prior_version: 33'1 first_divergent_update: 33'101 last_divergent_update: 33'101 2021-05-12T09:12:19.759 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:d858c035:::existing_1:head has no more recent entries in log 2021-05-12T09:12:19.759 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2021-05-12T09:12:19.760 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:d858c035:::existing_1:head olog_can_rollback_to: 33'106 2021-05-12T09:12:19.760 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:d858c035:::existing_1:head cannot rollback 33'101 (33'1) modify 2:d858c035:::existing_1:head by client.4538.0:1 2021-05-12T09:09:54.139229+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:12:19.760 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2021-05-12T09:12:19.760 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:d858c035:::existing_1:head prior_version 33'1 <= info.log_tail 33'84 2021-05-12T09:12:19.761 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 20 _merge_object_divergent_entries: merging hoid 2:da59f6e7:::newobject_1:head entries: 33'103 (0'0) modify 2:da59f6e7:::newobject_1:head by client.4544.0:1 2021-05-12T09:09:54.153392+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T09:12:19.761 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 20 _merge_object_divergent_entries: keeping 33'103 (0'0) modify 2:da59f6e7:::newobject_1:head by client.4544.0:1 2021-05-12T09:09:54.153392+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T09:12:19.761 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T09:12:19.761 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 33'103 last_divergent_update: 33'103 2021-05-12T09:12:19.762 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:da59f6e7:::newobject_1:head has no more recent entries in log 2021-05-12T09:12:19.762 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.157+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2021-05-12T09:12:19.762 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.158+0000 7f4ef84c3700 20 _merge_object_divergent_entries: merging hoid 2:97db2a0f:::existing_2:head entries: 33'102 (33'2) modify 2:97db2a0f:::existing_2:head by client.4539.0:1 2021-05-12T09:09:54.142211+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:12:19.762 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.158+0000 7f4ef84c3700 20 _merge_object_divergent_entries: keeping 33'102 (33'2) modify 2:97db2a0f:::existing_2:head by client.4539.0:1 2021-05-12T09:09:54.142211+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:12:19.763 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.158+0000 7f4ef84c3700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T09:12:19.763 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.158+0000 7f4ef84c3700 10 _merge_object_divergent_entries: hoid 2:97db2a0f:::existing_2:head prior_version: 33'2 first_divergent_update: 33'102 last_divergent_update: 33'102 2021-05-12T09:12:19.763 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.158+0000 7f4ef84c3700 10 _merge_object_divergent_entries: hoid 2:97db2a0f:::existing_2:head has no more recent entries in log 2021-05-12T09:12:19.763 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.158+0000 7f4ef84c3700 10 _merge_object_divergent_entries: hoid 2:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2021-05-12T09:12:19.764 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.158+0000 7f4ef84c3700 10 _merge_object_divergent_entries: hoid 2:97db2a0f:::existing_2:head olog_can_rollback_to: 33'106 2021-05-12T09:12:19.764 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.158+0000 7f4ef84c3700 10 _merge_object_divergent_entries: hoid 2:97db2a0f:::existing_2:head cannot rollback 33'102 (33'2) modify 2:97db2a0f:::existing_2:head by client.4539.0:1 2021-05-12T09:09:54.142211+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:12:19.764 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.158+0000 7f4ef84c3700 10 _merge_object_divergent_entries: hoid 2:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2021-05-12T09:12:19.765 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.158+0000 7f4ef84c3700 10 _merge_object_divergent_entries: hoid 2:97db2a0f:::existing_2:head prior_version 33'2 <= info.log_tail 33'84 2021-05-12T09:12:19.765 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.161+0000 7f4ef6cc0700 20 _merge_object_divergent_entries: merging hoid 2:70b3bf12:::existing_4:head entries: 33'106 (33'4) delete 2:70b3bf12:::existing_4:head by client.4551.0:1 2021-05-12T09:09:54.159344+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T09:12:19.766 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.161+0000 7f4ef6cc0700 20 _merge_object_divergent_entries: keeping 33'106 (33'4) delete 2:70b3bf12:::existing_4:head by client.4551.0:1 2021-05-12T09:09:54.159344+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T09:12:19.766 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.161+0000 7f4ef6cc0700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T09:12:19.767 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.161+0000 7f4ef6cc0700 10 _merge_object_divergent_entries: hoid 2:70b3bf12:::existing_4:head prior_version: 33'4 first_divergent_update: 33'106 last_divergent_update: 33'106 2021-05-12T09:12:19.767 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.161+0000 7f4ef6cc0700 10 _merge_object_divergent_entries: more recent entry found: 51'101 (33'4) modify 2:70b3bf12:::existing_4:head by client.4664.0:1 2021-05-12T09:10:53.268103+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2021-05-12T09:12:19.767 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.162+0000 7f4ef7cc2700 20 _merge_object_divergent_entries: merging hoid 2:8881e6d8:::existing_3:head entries: 33'105 (33'3) delete 2:8881e6d8:::existing_3:head by client.4549.0:1 2021-05-12T09:09:54.157502+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T09:12:19.767 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.162+0000 7f4ef7cc2700 20 _merge_object_divergent_entries: keeping 33'105 (33'3) delete 2:8881e6d8:::existing_3:head by client.4549.0:1 2021-05-12T09:09:54.157502+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T09:12:19.768 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.162+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2021-05-12T09:12:19.778 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.162+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:8881e6d8:::existing_3:head prior_version: 33'3 first_divergent_update: 33'105 last_divergent_update: 33'105 2021-05-12T09:12:19.779 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.162+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:8881e6d8:::existing_3:head has no more recent entries in log 2021-05-12T09:12:19.779 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.162+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2021-05-12T09:12:19.780 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.162+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:8881e6d8:::existing_3:head olog_can_rollback_to: 33'106 2021-05-12T09:12:19.781 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.162+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:8881e6d8:::existing_3:head cannot rollback 33'105 (33'3) delete 2:8881e6d8:::existing_3:head by client.4549.0:1 2021-05-12T09:09:54.157502+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-05-12T09:12:19.781 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.162+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2021-05-12T09:12:19.781 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.162+0000 7f4ef7cc2700 10 _merge_object_divergent_entries: hoid 2:8881e6d8:::existing_3:head prior_version 33'3 <= info.log_tail 33'84 2021-05-12T09:12:19.782 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.306+0000 7f4ef84c3700 20 _merge_object_divergent_entries: merging hoid 2:75ac5091:::newobject_2:head entries: 33'104 (0'0) modify 2:75ac5091:::newobject_2:head by client.4545.0:1 2021-05-12T09:09:54.153568+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T09:12:19.782 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.306+0000 7f4ef84c3700 20 _merge_object_divergent_entries: keeping 33'104 (0'0) modify 2:75ac5091:::newobject_2:head by client.4545.0:1 2021-05-12T09:09:54.153568+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T09:12:19.783 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.306+0000 7f4ef84c3700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T09:12:19.783 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.306+0000 7f4ef84c3700 10 _merge_object_divergent_entries: hoid 2:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 33'104 last_divergent_update: 33'104 2021-05-12T09:12:19.783 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.306+0000 7f4ef84c3700 10 _merge_object_divergent_entries: hoid 2:75ac5091:::newobject_2:head has no more recent entries in log 2021-05-12T09:12:19.783 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:11:38.306+0000 7f4ef84c3700 10 _merge_object_divergent_entries: hoid 2:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2021-05-12T09:12:19.785 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-05-12T09:12:19.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T09:12:19.796 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-05-12T09:12:19.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T09:12:19.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2021-05-12T09:12:19.815 INFO:tasks.workunit.client.0.smithi028.stdout:success 2021-05-12T09:12:19.816 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.29719.out 2021-05-12T09:12:19.816 INFO:tasks.workunit.client.0.smithi028.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.29719.out': No such file or directory 2021-05-12T09:12:19.817 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2021-05-12T09:12:19.817 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-05-12T09:12:19.817 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-05-12T09:12:20.301 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'test' does not exist 2021-05-12T09:12:20.310 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2021-05-12T09:12:20.311 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:12:20.311 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:12:20.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:12:20.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:12:20.312 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:12:20.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:12:20.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-05-12T09:12:20.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T09:12:20.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:12:20.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T09:12:20.421 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:12:20.421 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:12:20.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:12:20.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:12:20.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:12:20.424 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:12:20.424 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:12:20.425 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:12:20.425 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:12:20.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:12:20.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:12:20.427 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:12:20.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:12:20.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:12:20.429 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:12:20.429 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:12:20.429 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:12:20.430 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:12:20.430 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T09:12:20.457 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:12:20.458 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:12:20.458 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:12:20.458 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29719 2021-05-12T09:12:20.460 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:12:20.460 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:12:20.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-05-12T09:12:20.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-05-12T09:12:20.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-05-12T09:12:20.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-05-12T09:12:20.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T09:12:20.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:12:20.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T09:12:20.462 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:12:20.462 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:12:20.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:12:20.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:12:20.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:12:20.465 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:12:20.465 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:12:20.465 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:12:20.466 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:12:20.466 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:12:20.466 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:12:20.466 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:12:20.466 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:12:20.467 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:12:20.467 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:12:20.469 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:12:20.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:12:20.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:12:20.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T09:12:20.470 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:12:20.470 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:12:20.470 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:12:20.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29719 2021-05-12T09:12:20.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:12:20.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:12:20.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-05-12T09:12:20.473 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:12:20.474 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:12:20.474 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:12:20.474 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.29719 2021-05-12T09:12:20.474 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:12:20.474 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T09:12:20.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:12:20.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-05-12T09:12:20.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2021-05-12T09:12:20.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2021-05-12T09:12:20.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2021-05-12T09:12:20.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2021-05-12T09:12:20.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2021-05-12T09:12:20.476 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2021-05-12T09:12:20.478 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2021-05-12T09:12:20.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2021-05-12T09:12:20.478 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:12:20.479 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:12:20.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2021-05-12T09:12:20.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-05-12T09:12:20.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:12:20.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:12:20.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:12:20.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-05-12T09:12:20.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-05-12T09:12:20.851 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:12:20.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:12:20.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:12:20.852 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:12:20.852 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:12:20.853 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:12:20.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:12:20.853 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:12:21.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:12:21.079 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:12:21.079 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:12:21.079 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:12:21.080 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:12:21.081 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:12:21.081 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:12:21.081 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:12:21.082 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:12:21.082 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:12:21.082 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:12:21.082 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:12:21.083 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-mon.a.asok 2021-05-12T09:12:21.083 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:12:21.083 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29719/ceph-mon.a.asok config get fsid 2021-05-12T09:12:21.190 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:12:21.191 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:12:21.191 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:12:21.191 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:12:21.191 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:12:21.192 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:12:21.192 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:12:21.192 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:12:21.192 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:12:21.193 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:12:21.193 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:12:21.193 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-mon.a.asok 2021-05-12T09:12:21.193 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:12:21.194 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.29719/ceph-mon.a.asok config get mon_host 2021-05-12T09:12:21.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2021-05-12T09:12:21.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-05-12T09:12:21.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:12:21.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:12:21.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:12:21.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-05-12T09:12:21.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:12:22.183 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:12:22.183 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:12:22.184 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:12:22.184 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:12:22.184 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:12:22.184 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:12:22.185 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:12:22.185 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:12:22.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:12:22.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-05-12T09:12:22.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2021-05-12T09:12:22.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T09:12:22.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:12:22.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:12:22.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:12:22.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-05-12T09:12:22.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:12:22.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:12:22.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:12:22.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:12:22.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T09:12:22.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T09:12:22.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:12:22.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:12:22.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:12:22.220 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:12:22.220 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:12:22.220 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:12:22.220 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:12:22.221 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:12:22.221 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:12:22.221 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:12:22.221 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:12:22.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:12:22.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:12:22.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:12:22.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:12:22.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:12:22.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:12:22.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:12:22.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:12:22.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:12:22.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:12:22.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/0 2021-05-12T09:12:22.224 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:12:22.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=211d7d1b-7ece-4928-932a-3aa488ac19a5 2021-05-12T09:12:22.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 211d7d1b-7ece-4928-932a-3aa488ac19a5' 2021-05-12T09:12:22.226 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 211d7d1b-7ece-4928-932a-3aa488ac19a5 2021-05-12T09:12:22.226 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:12:22.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQD2m5tgi08eDhAAX6e96POA/UywuaqD0zcvSg== 2021-05-12T09:12:22.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQD2m5tgi08eDhAAX6e96POA/UywuaqD0zcvSg=="}' 2021-05-12T09:12:22.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 211d7d1b-7ece-4928-932a-3aa488ac19a5 -i td/divergent-priors/0/new.json 2021-05-12T09:12:22.569 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:12:22.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/0/new.json 2021-05-12T09:12:22.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD2m5tgi08eDhAAX6e96POA/UywuaqD0zcvSg== --osd-uuid 211d7d1b-7ece-4928-932a-3aa488ac19a5 2021-05-12T09:12:22.613 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:22.613+0000 7f21220800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:22.614 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:22.614+0000 7f21220800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:22.616 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:22.616+0000 7f21220800c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:22.616 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:22.617+0000 7f21220800c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T09:12:22.617 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:22.617+0000 7f21220800c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T09:12:22.617 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:22.617+0000 7f21220800c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-05-12T09:12:22.659 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:22.659+0000 7f21220800c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-05-12T09:12:26.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-05-12T09:12:26.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:12:26.952 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:12:26.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:12:26.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:12:27.215 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:12:27.216 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:12:27.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:12:27.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:12:27.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:12:27.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:12:27.245 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:27.245+0000 7f950e9000c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:27.247 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:27.247+0000 7f950e9000c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:27.249 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:27.249+0000 7f950e9000c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:27.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:12:27.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:12:27.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:12:27.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:12:27.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:12:27.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:12:27.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:12:27.609 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:12:27.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:12:27.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:12:27.771 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:27.771+0000 7f950e9000c0 -1 Falling back to public interface 2021-05-12T09:12:28.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:12:29.010 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:12:29.010 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:12:29.011 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:12:29.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:12:29.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:12:29.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:12:29.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:12:29.978 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:29.977+0000 7f950e9000c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:12:30.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:12:30.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:12:30.406 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:12:30.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:12:30.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:12:30.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:12:30.806 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/69763,v1:127.0.0.1:6803/69763] [v2:127.0.0.1:6804/69763,v1:127.0.0.1:6805/69763] exists,up 211d7d1b-7ece-4928-932a-3aa488ac19a5 2021-05-12T09:12:30.811 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:12:30.811 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:12:30.811 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:12:30.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-05-12T09:12:30.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2021-05-12T09:12:30.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T09:12:30.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:12:30.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:12:30.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:12:30.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-05-12T09:12:30.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:12:30.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:12:30.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:12:30.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:12:30.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T09:12:30.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T09:12:30.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:12:30.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:12:30.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:12:30.816 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:12:30.816 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:12:30.816 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:12:30.816 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:12:30.817 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:12:30.817 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:12:30.817 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:12:30.817 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:12:30.817 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:12:30.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:12:30.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:12:30.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:12:30.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:12:30.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:12:30.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:12:30.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:12:30.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:12:30.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:12:30.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/1 2021-05-12T09:12:30.820 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:12:30.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=860f4a3c-1b2f-41ed-b1b5-8d8eaa20b04b 2021-05-12T09:12:30.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 860f4a3c-1b2f-41ed-b1b5-8d8eaa20b04b' 2021-05-12T09:12:30.821 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 860f4a3c-1b2f-41ed-b1b5-8d8eaa20b04b 2021-05-12T09:12:30.821 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:12:30.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQD+m5tgH11yMRAA6FYdUR+1o8XJunuTRTyTRA== 2021-05-12T09:12:30.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQD+m5tgH11yMRAA6FYdUR+1o8XJunuTRTyTRA=="}' 2021-05-12T09:12:30.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 860f4a3c-1b2f-41ed-b1b5-8d8eaa20b04b -i td/divergent-priors/1/new.json 2021-05-12T09:12:31.342 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:12:31.353 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/1/new.json 2021-05-12T09:12:31.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD+m5tgH11yMRAA6FYdUR+1o8XJunuTRTyTRA== --osd-uuid 860f4a3c-1b2f-41ed-b1b5-8d8eaa20b04b 2021-05-12T09:12:31.385 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:31.385+0000 7ff656aa30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:31.387 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:31.387+0000 7ff656aa30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:31.388 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:31.388+0000 7ff656aa30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:31.389 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:31.389+0000 7ff656aa30c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T09:12:31.389 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:31.389+0000 7ff656aa30c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T09:12:31.389 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:31.389+0000 7ff656aa30c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-05-12T09:12:31.488 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:31.488+0000 7ff656aa30c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-05-12T09:12:35.806 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-05-12T09:12:35.806 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:12:35.807 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:12:35.807 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:12:35.807 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:12:36.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:12:36.339 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:12:36.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:12:36.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:12:36.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:12:36.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:12:36.369 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:36.367+0000 7f99253550c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:36.386 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:36.385+0000 7f99253550c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:36.388 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:36.387+0000 7f99253550c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:36.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:12:36.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:12:36.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:12:36.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:12:36.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:12:36.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:12:36.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:12:36.725 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:12:36.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:12:36.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:12:36.904 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:36.903+0000 7f99253550c0 -1 Falling back to public interface 2021-05-12T09:12:37.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:12:38.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:12:38.122 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:12:38.123 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:12:38.123 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:12:38.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:12:38.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:12:38.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:12:39.147 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:39.146+0000 7f99253550c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:12:39.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:12:39.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:12:39.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:12:39.519 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:12:39.519 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:12:39.520 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:12:39.937 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/71148,v1:127.0.0.1:6811/71148] [v2:127.0.0.1:6812/71148,v1:127.0.0.1:6813/71148] exists,up 860f4a3c-1b2f-41ed-b1b5-8d8eaa20b04b 2021-05-12T09:12:39.941 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:12:39.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:12:39.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:12:39.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-05-12T09:12:39.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2021-05-12T09:12:39.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-05-12T09:12:39.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:12:39.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T09:12:39.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:12:39.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-05-12T09:12:39.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:12:39.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:12:39.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:12:39.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:12:39.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T09:12:39.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T09:12:39.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:12:39.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:12:39.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:12:39.948 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:12:39.950 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:12:39.950 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:12:39.950 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:12:39.950 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:12:39.951 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:12:39.951 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:12:39.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:12:39.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:12:39.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:12:39.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:12:39.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:12:39.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:12:39.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:12:39.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:12:39.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:12:39.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:12:39.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:12:39.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/divergent-priors/2 2021-05-12T09:12:39.954 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:12:39.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=7951a78e-ee8e-4c4f-a30b-05cc781094fa 2021-05-12T09:12:39.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 7951a78e-ee8e-4c4f-a30b-05cc781094fa' 2021-05-12T09:12:39.955 INFO:tasks.workunit.client.0.smithi028.stdout:add osd2 7951a78e-ee8e-4c4f-a30b-05cc781094fa 2021-05-12T09:12:39.955 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:12:39.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAHnJtgQ8lAORAAEwQznBC79peRhDdTaPUtWA== 2021-05-12T09:12:39.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAHnJtgQ8lAORAAEwQznBC79peRhDdTaPUtWA=="}' 2021-05-12T09:12:39.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 7951a78e-ee8e-4c4f-a30b-05cc781094fa -i td/divergent-priors/2/new.json 2021-05-12T09:12:40.496 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:12:40.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/divergent-priors/2/new.json 2021-05-12T09:12:40.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAHnJtgQ8lAORAAEwQznBC79peRhDdTaPUtWA== --osd-uuid 7951a78e-ee8e-4c4f-a30b-05cc781094fa 2021-05-12T09:12:40.539 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:40.538+0000 7fdf40d8f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:40.541 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:40.540+0000 7fdf40d8f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:40.543 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:40.541+0000 7fdf40d8f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:40.544 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:40.542+0000 7fdf40d8f0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T09:12:40.544 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:40.542+0000 7fdf40d8f0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T09:12:40.545 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:40.542+0000 7fdf40d8f0c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-05-12T09:12:40.626 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:40.626+0000 7fdf40d8f0c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-05-12T09:12:45.095 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-05-12T09:12:45.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:12:45.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T09:12:45.097 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd2 key to auth repository 2021-05-12T09:12:45.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:12:45.617 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:12:45.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T09:12:45.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:12:45.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:12:45.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:12:45.619 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:12:45.646 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:45.645+0000 7fb1ed7510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:45.648 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:45.647+0000 7fb1ed7510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:45.650 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:45.649+0000 7fb1ed7510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:12:46.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T09:12:46.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:12:46.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:12:46.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:12:46.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:12:46.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:12:46.029 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:12:46.029 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:12:46.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:12:46.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:12:46.169 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:46.168+0000 7fb1ed7510c0 -1 Falling back to public interface 2021-05-12T09:12:46.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:12:47.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:12:47.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:12:47.428 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:12:47.429 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:12:47.429 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:12:47.429 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:12:47.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:12:48.361 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:12:48.360+0000 7fb1ed7510c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T09:12:48.825 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:12:48.826 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:12:48.826 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:12:48.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:12:48.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:12:48.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:12:49.234 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/72533,v1:127.0.0.1:6819/72533] [v2:127.0.0.1:6820/72533,v1:127.0.0.1:6821/72533] exists,up 7951a78e-ee8e-4c4f-a30b-05cc781094fa 2021-05-12T09:12:49.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:12:49.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:12:49.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:12:49.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2021-05-12T09:12:50.407 INFO:tasks.workunit.client.0.smithi028.stderr:noout is set 2021-05-12T09:12:50.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2021-05-12T09:12:51.477 INFO:tasks.workunit.client.0.smithi028.stderr:noin is set 2021-05-12T09:12:51.487 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2021-05-12T09:12:52.318 INFO:tasks.workunit.client.0.smithi028.stderr:nodown is set 2021-05-12T09:12:52.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2021-05-12T09:12:52.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: create_ec_pool: local pool_name=test 2021-05-12T09:12:52.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: create_ec_pool: shift 2021-05-12T09:12:52.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: create_ec_pool: local allow_overwrites=true 2021-05-12T09:12:52.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: shift 2021-05-12T09:12:52.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2021-05-12T09:12:52.925 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: create_ec_pool: create_pool test 1 1 erasure myprofile 2021-05-12T09:12:52.925 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2021-05-12T09:12:53.838 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'test' already exists 2021-05-12T09:12:53.847 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:12:54.848 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: create_ec_pool: '[' true = true ']' 2021-05-12T09:12:54.849 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2021-05-12T09:12:56.034 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 2 allow_ec_overwrites to true 2021-05-12T09:12:56.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: create_ec_pool: wait_for_clean 2021-05-12T09:12:56.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:12:56.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:12:56.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:12:56.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:12:56.046 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T09:12:56.046 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:12:56.046 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:12:56.047 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:12:56.047 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:12:56.048 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:12:56.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:12:56.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:12:56.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:12:56.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:12:56.141 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:12:56.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:12:56.529 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:12:56.529 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:12:56.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:12:56.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:12:56.530 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:12:56.669 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836488 2021-05-12T09:12:56.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836488 2021-05-12T09:12:56.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488' 2021-05-12T09:12:56.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:12:56.670 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:12:56.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672966 2021-05-12T09:12:56.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672966 2021-05-12T09:12:56.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488 1-42949672966' 2021-05-12T09:12:56.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:12:56.810 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:12:56.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509444 2021-05-12T09:12:56.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509444 2021-05-12T09:12:56.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-64424509444' 2021-05-12T09:12:56.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:12:56.952 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836488 2021-05-12T09:12:56.952 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:12:56.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:12:56.953 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836488 2021-05-12T09:12:56.953 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:12:56.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836488 2021-05-12T09:12:56.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-05-12T09:12:56.955 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836488 2021-05-12T09:12:56.955 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:12:57.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836488 2021-05-12T09:12:57.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:12:57.360 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672966 2021-05-12T09:12:57.360 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:12:57.361 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:12:57.361 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672966 2021-05-12T09:12:57.362 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:12:57.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672966 2021-05-12T09:12:57.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-05-12T09:12:57.363 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672966 2021-05-12T09:12:57.364 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:12:57.748 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672966 -lt 42949672966 2021-05-12T09:12:57.748 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:12:57.749 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-05-12T09:12:57.749 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:12:57.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:12:57.750 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509444 2021-05-12T09:12:57.750 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:12:57.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509444 2021-05-12T09:12:57.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-05-12T09:12:57.752 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509444 2021-05-12T09:12:57.752 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:12:58.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509444 -lt 64424509444 2021-05-12T09:12:58.152 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:12:58.152 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:12:58.152 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:12:58.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:12:58.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:12:58.677 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:12:58.678 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:12:58.678 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:12:58.678 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:12:58.678 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:12:58.679 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:12:59.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T09:12:59.053 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:12:59.053 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:12:59.054 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:12:59.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T09:12:59.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:12:59.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:12:59.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: create_ec_pool: return 0 2021-05-12T09:12:59.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2021-05-12T09:12:59.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:12:59.575 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:12:59.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:12:59.959 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:12:59.960 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:12:59.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:12:59.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:12:59.961 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:13:00.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836490 2021-05-12T09:13:00.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836490 2021-05-12T09:13:00.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490' 2021-05-12T09:13:00.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:13:00.110 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:13:00.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672968 2021-05-12T09:13:00.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672968 2021-05-12T09:13:00.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2021-05-12T09:13:00.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:13:00.250 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:13:00.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509447 2021-05-12T09:13:00.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509447 2021-05-12T09:13:00.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509447' 2021-05-12T09:13:00.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:13:00.392 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-05-12T09:13:00.392 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:13:00.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:13:00.393 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836490 2021-05-12T09:13:00.393 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:13:00.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836490 2021-05-12T09:13:00.394 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836490 2021-05-12T09:13:00.395 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-05-12T09:13:00.395 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:13:00.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836490 2021-05-12T09:13:00.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:13:01.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:13:01.783 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:13:02.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836490 -lt 21474836490 2021-05-12T09:13:02.187 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:13:02.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2021-05-12T09:13:02.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:13:02.189 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:13:02.190 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672968 2021-05-12T09:13:02.190 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:13:02.190 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672968 2021-05-12T09:13:02.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-05-12T09:13:02.191 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672968 2021-05-12T09:13:02.191 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:13:02.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672969 -lt 42949672968 2021-05-12T09:13:02.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:13:02.582 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509447 2021-05-12T09:13:02.582 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:13:02.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:13:02.584 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509447 2021-05-12T09:13:02.584 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:13:02.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509447 2021-05-12T09:13:02.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2021-05-12T09:13:02.585 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509447 2021-05-12T09:13:02.586 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:13:02.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509447 -lt 64424509447 2021-05-12T09:13:02.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2021-05-12T09:13:02.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:13:02.980 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:13:02.980 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:13:02.980 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:13:02.980 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T09:13:02.981 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:13:02.981 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:13:02.981 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:13:02.981 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:13:02.982 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:13:03.074 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:13:03.075 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:13:03.075 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:13:03.075 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:13:03.075 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:13:03.459 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:13:03.459 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:13:03.460 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:13:03.460 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:13:03.460 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:13:03.460 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:13:03.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836493 2021-05-12T09:13:03.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836493 2021-05-12T09:13:03.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836493' 2021-05-12T09:13:03.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:13:03.610 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:13:03.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672971 2021-05-12T09:13:03.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672971 2021-05-12T09:13:03.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836493 1-42949672971' 2021-05-12T09:13:03.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:13:03.751 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:13:03.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509449 2021-05-12T09:13:03.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509449 2021-05-12T09:13:03.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509449' 2021-05-12T09:13:03.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:13:03.891 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836493 2021-05-12T09:13:03.891 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:13:03.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:13:03.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836493 2021-05-12T09:13:03.893 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:13:03.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836493 2021-05-12T09:13:03.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2021-05-12T09:13:03.894 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836493 2021-05-12T09:13:03.894 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:13:04.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836491 -lt 21474836493 2021-05-12T09:13:04.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:13:05.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:13:05.283 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:13:05.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836493 -lt 21474836493 2021-05-12T09:13:05.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:13:05.677 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672971 2021-05-12T09:13:05.677 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:13:05.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:13:05.678 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672971 2021-05-12T09:13:05.679 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:13:05.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672971 2021-05-12T09:13:05.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2021-05-12T09:13:05.680 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672971 2021-05-12T09:13:05.681 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:13:06.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672971 -lt 42949672971 2021-05-12T09:13:06.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:13:06.087 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509449 2021-05-12T09:13:06.087 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:13:06.088 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:13:06.088 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509449 2021-05-12T09:13:06.088 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:13:06.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509449 2021-05-12T09:13:06.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2021-05-12T09:13:06.089 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509449 2021-05-12T09:13:06.090 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:13:06.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509450 -lt 64424509449 2021-05-12T09:13:06.479 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:13:06.480 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:13:06.480 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:13:06.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:13:06.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:13:06.985 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:13:06.985 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:13:06.985 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:13:06.985 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:13:06.985 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:13:06.986 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:13:07.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T09:13:07.377 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:13:07.378 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:13:07.378 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:13:07.891 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T09:13:07.891 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:13:07.891 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:13:07.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2021-05-12T09:13:07.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2021-05-12T09:13:08.256 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:13:08.266 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=2 2021-05-12T09:13:08.267 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 2' 2021-05-12T09:13:08.268 INFO:tasks.workunit.client.0.smithi028.stdout:primary and soon to be divergent is 2 2021-05-12T09:13:08.268 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T09:13:08.629 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:13:08.630 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T09:12:55.285903+0000 0'0 31:12 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:12:53.510061+0000 0'0 2021-05-12T09:12:53.510061+0000 0 2021-05-12T09:13:08.630 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:12:50.160220+0000 32'62 32:107 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:12:49.062969+0000 0'0 2021-05-12T09:12:49.062969+0000 0 2021-05-12T09:13:08.630 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:13:08.630 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:13:08.631 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:13:08.641 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2021-05-12T09:13:08.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-05-12T09:13:08.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 2 ']' 2021-05-12T09:13:08.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2021-05-12T09:13:08.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-05-12T09:13:08.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 2 ']' 2021-05-12T09:13:08.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 1' 2021-05-12T09:13:08.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-05-12T09:13:08.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 2 ']' 2021-05-12T09:13:08.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2021-05-12T09:13:08.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2021-05-12T09:13:08.644 INFO:tasks.workunit.client.0.smithi028.stdout:writing initial objects 2021-05-12T09:13:08.645 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2021-05-12T09:13:08.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:08.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2021-05-12T09:13:08.758 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:08.759 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2021-05-12T09:13:08.994 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:08.994 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2021-05-12T09:13:09.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:09.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2021-05-12T09:13:09.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:09.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2021-05-12T09:13:09.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:09.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2021-05-12T09:13:09.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:09.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2021-05-12T09:13:10.024 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:10.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2021-05-12T09:13:10.242 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:10.242 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2021-05-12T09:13:10.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:10.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2021-05-12T09:13:10.636 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:10.636 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2021-05-12T09:13:10.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:10.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2021-05-12T09:13:11.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:11.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2021-05-12T09:13:11.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:11.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2021-05-12T09:13:11.449 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:11.449 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2021-05-12T09:13:11.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:11.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2021-05-12T09:13:11.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:11.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2021-05-12T09:13:12.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:12.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2021-05-12T09:13:12.320 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:12.321 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2021-05-12T09:13:12.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:12.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2021-05-12T09:13:12.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:12.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2021-05-12T09:13:12.891 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:12.891 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2021-05-12T09:13:13.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:13.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2021-05-12T09:13:13.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:13.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2021-05-12T09:13:13.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:13.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2021-05-12T09:13:13.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:13.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2021-05-12T09:13:13.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:13.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2021-05-12T09:13:14.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:14.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2021-05-12T09:13:14.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:14.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2021-05-12T09:13:14.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:14.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2021-05-12T09:13:14.668 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:14.668 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2021-05-12T09:13:14.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:14.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2021-05-12T09:13:14.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:14.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2021-05-12T09:13:15.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:15.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2021-05-12T09:13:15.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:15.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2021-05-12T09:13:15.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:15.640 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2021-05-12T09:13:15.843 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:15.843 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2021-05-12T09:13:16.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:16.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2021-05-12T09:13:16.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:16.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2021-05-12T09:13:16.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:16.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2021-05-12T09:13:16.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:16.648 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2021-05-12T09:13:16.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:16.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2021-05-12T09:13:16.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:16.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2021-05-12T09:13:17.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:17.194 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2021-05-12T09:13:17.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:17.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2021-05-12T09:13:17.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:17.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2021-05-12T09:13:17.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:17.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2021-05-12T09:13:17.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:17.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2021-05-12T09:13:18.215 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:18.215 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2021-05-12T09:13:18.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:18.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2021-05-12T09:13:18.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:18.626 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2021-05-12T09:13:18.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:18.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2021-05-12T09:13:18.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:18.987 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2021-05-12T09:13:19.171 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:19.171 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2021-05-12T09:13:19.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:19.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2021-05-12T09:13:19.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:19.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2021-05-12T09:13:19.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:19.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2021-05-12T09:13:19.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:19.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2021-05-12T09:13:20.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:20.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2021-05-12T09:13:20.361 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:20.361 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2021-05-12T09:13:20.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:20.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2021-05-12T09:13:20.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:20.730 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2021-05-12T09:13:20.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:20.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2021-05-12T09:13:21.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:21.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2021-05-12T09:13:21.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:21.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2021-05-12T09:13:21.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:21.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2021-05-12T09:13:21.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:21.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2021-05-12T09:13:21.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:21.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2021-05-12T09:13:22.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:22.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2021-05-12T09:13:22.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:22.298 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2021-05-12T09:13:22.491 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:22.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2021-05-12T09:13:22.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:22.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2021-05-12T09:13:22.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:22.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2021-05-12T09:13:23.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:23.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2021-05-12T09:13:23.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:23.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2021-05-12T09:13:23.446 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:23.446 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2021-05-12T09:13:23.622 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:23.622 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2021-05-12T09:13:23.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:23.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2021-05-12T09:13:24.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:24.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2021-05-12T09:13:24.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:24.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2021-05-12T09:13:24.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:24.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2021-05-12T09:13:24.619 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:24.620 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2021-05-12T09:13:24.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:24.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2021-05-12T09:13:24.971 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:24.972 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2021-05-12T09:13:25.172 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:25.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2021-05-12T09:13:25.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:25.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2021-05-12T09:13:25.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:25.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2021-05-12T09:13:25.976 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:25.976 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2021-05-12T09:13:26.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:26.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2021-05-12T09:13:26.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:26.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2021-05-12T09:13:26.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:26.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2021-05-12T09:13:26.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:26.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2021-05-12T09:13:26.857 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:26.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2021-05-12T09:13:27.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:27.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2021-05-12T09:13:27.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:27.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2021-05-12T09:13:27.486 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:27.487 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2021-05-12T09:13:27.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:27.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2021-05-12T09:13:27.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:27.863 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2021-05-12T09:13:28.076 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:28.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2021-05-12T09:13:28.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-05-12T09:13:28.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2021-05-12T09:13:28.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T09:13:28.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2021-05-12T09:13:28.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:13:28.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:13:28.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:13:28.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:13:28.476 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T09:13:28.477 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:13:28.477 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:13:28.478 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:13:28.478 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:13:28.478 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:13:28.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:13:28.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:13:28.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:13:28.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:13:28.546 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:13:28.933 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:13:28.934 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:13:28.934 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:13:28.934 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:13:28.935 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:13:28.935 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:13:29.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836500 2021-05-12T09:13:29.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836500 2021-05-12T09:13:29.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836500' 2021-05-12T09:13:29.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:13:29.088 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:13:29.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672978 2021-05-12T09:13:29.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672978 2021-05-12T09:13:29.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836500 1-42949672978' 2021-05-12T09:13:29.229 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:13:29.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:13:29.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509456 2021-05-12T09:13:29.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509456 2021-05-12T09:13:29.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-64424509456' 2021-05-12T09:13:29.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:13:29.369 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836500 2021-05-12T09:13:29.370 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:13:29.370 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:13:29.371 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836500 2021-05-12T09:13:29.371 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:13:29.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836500 2021-05-12T09:13:29.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2021-05-12T09:13:29.372 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836500 2021-05-12T09:13:29.373 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:13:29.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836498 -lt 21474836500 2021-05-12T09:13:29.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:13:30.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:13:30.756 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:13:31.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836500 -lt 21474836500 2021-05-12T09:13:31.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:13:31.202 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672978 2021-05-12T09:13:31.202 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:13:31.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:13:31.203 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672978 2021-05-12T09:13:31.203 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:13:31.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672978 2021-05-12T09:13:31.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2021-05-12T09:13:31.205 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672978 2021-05-12T09:13:31.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:13:31.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672979 -lt 42949672978 2021-05-12T09:13:31.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:13:31.591 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509456 2021-05-12T09:13:31.591 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:13:31.592 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:13:31.592 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509456 2021-05-12T09:13:31.592 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:13:31.593 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509456 2021-05-12T09:13:31.593 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2021-05-12T09:13:31.594 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509456 2021-05-12T09:13:31.594 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:13:31.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509457 -lt 64424509456 2021-05-12T09:13:31.979 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:13:31.980 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:13:31.980 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:13:32.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:13:32.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:13:32.497 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:13:32.497 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:13:32.497 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:13:32.498 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:13:32.498 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:13:32.498 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:13:32.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T09:13:32.873 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:13:32.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:13:32.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:13:33.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T09:13:33.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:13:33.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:13:33.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2021-05-12T09:13:33.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local poolname=test 2021-05-12T09:13:33.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: local objectname=existing_1 2021-05-12T09:13:33.381 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: ceph --format json osd map test existing_1 2021-05-12T09:13:33.381 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: jq -r .pgid 2021-05-12T09:13:33.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=2.0 2021-05-12T09:13:33.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 1' 2021-05-12T09:13:33.772 INFO:tasks.workunit.client.0.smithi028.stdout:blackholing osds 0 1 2021-05-12T09:13:33.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T09:13:34.137 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:13:34.137 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-05-12T09:12:55.285903+0000 32'100 32:112 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:12:53.510061+0000 0'0 2021-05-12T09:12:53.510061+0000 0 2021-05-12T09:13:34.137 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:12:50.160220+0000 32'62 32:107 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:12:49.062969+0000 0'0 2021-05-12T09:12:49.062969+0000 0 2021-05-12T09:13:34.138 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:13:34.138 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:13:34.139 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:13:34.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2021-05-12T09:13:34.150 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2021-05-12T09:13:34.150 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T09:13:34.150 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T09:13:34.150 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:13:34.151 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:13:34.151 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:13:34.151 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.0.asok 2021-05-12T09:13:34.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T09:13:34.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.0.asok config set objectstore_blackhole 1 2021-05-12T09:13:34.246 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:13:34.247 INFO:tasks.workunit.client.0.smithi028.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T09:13:34.247 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:13:34.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2021-05-12T09:13:34.256 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.1 2021-05-12T09:13:34.257 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T09:13:34.257 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T09:13:34.257 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:13:34.257 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:13:34.257 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:13:34.258 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.1.asok 2021-05-12T09:13:34.258 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T09:13:34.258 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.1.asok config set objectstore_blackhole 1 2021-05-12T09:13:34.352 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:13:34.352 INFO:tasks.workunit.client.0.smithi028.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-05-12T09:13:34.353 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:13:34.360 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2021-05-12T09:13:34.361 INFO:tasks.workunit.client.0.smithi028.stdout:writing divergent object 2021-05-12T09:13:34.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2021-05-12T09:13:34.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2021-05-12T09:13:35.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2021-05-12T09:13:35.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2021-05-12T09:13:35.561 INFO:tasks.workunit.client.0.smithi028.stdout:created pool test snap snap1 2021-05-12T09:13:35.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2021-05-12T09:13:35.565 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2021-05-12T09:13:35.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2021-05-12T09:13:45.566 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2021-05-12T09:13:45.569 INFO:tasks.workunit.client.0.smithi028.stderr:rados: no process found 2021-05-12T09:13:45.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2021-05-12T09:13:45.570 INFO:tasks.workunit.client.0.smithi028.stdout:killing all the osds 2021-05-12T09:13:45.571 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T09:13:45.945 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:13:45.945 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 101 0 0 0 0 48783 0 0 14 14 active+clean 2021-05-12T09:12:55.285903+0000 33'104 33:116 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:12:53.510061+0000 0'0 2021-05-12T09:12:53.510061+0000 0 2021-05-12T09:13:45.945 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:12:50.160220+0000 32'62 32:108 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:12:49.062969+0000 0'0 2021-05-12T09:12:49.062969+0000 0 2021-05-12T09:13:45.946 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:13:45.946 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:13:45.947 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:13:45.957 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2021-05-12T09:13:45.958 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:13:45.958 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:13:45.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:13:45.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:13:45.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:13:46.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:13:46.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-05-12T09:13:46.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2021-05-12T09:13:46.633 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already down. 2021-05-12T09:13:46.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-05-12T09:13:46.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2021-05-12T09:13:47.175 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already down. 2021-05-12T09:13:47.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-05-12T09:13:47.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2021-05-12T09:13:47.759 INFO:tasks.workunit.client.0.smithi028.stderr:osd.2 is already down. 2021-05-12T09:13:47.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2021-05-12T09:13:47.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2021-05-12T09:13:48.297 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already out. 2021-05-12T09:13:48.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2021-05-12T09:13:48.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.1 2021-05-12T09:13:48.837 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already out. 2021-05-12T09:13:48.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 1' 2021-05-12T09:13:48.847 INFO:tasks.workunit.client.0.smithi028.stdout:bringing up non_divergent 0 1 2021-05-12T09:13:48.848 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T09:13:49.212 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:13:49.213 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 101 0 0 0 0 48783 0 0 14 14 stale+active+clean 2021-05-12T09:12:55.285903+0000 33'104 33:116 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:12:53.510061+0000 0'0 2021-05-12T09:12:53.510061+0000 0 2021-05-12T09:13:49.213 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 stale+active+clean 2021-05-12T09:12:50.160220+0000 32'62 32:108 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:12:49.062969+0000 0'0 2021-05-12T09:12:49.062969+0000 0 2021-05-12T09:13:49.213 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:13:49.213 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:13:49.214 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:13:49.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2021-05-12T09:13:49.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2021-05-12T09:13:49.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:13:49.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:13:49.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T09:13:49.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:13:49.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T09:13:49.229 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:13:49.229 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:13:49.229 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:13:49.229 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:13:49.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T09:13:49.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T09:13:49.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:13:49.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:13:49.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:13:49.231 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:13:49.231 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:13:49.231 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:13:49.231 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:13:49.234 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:13:49.234 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:13:49.235 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:13:49.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:13:49.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:13:49.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:13:49.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:13:49.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:13:49.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:13:49.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:13:49.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:13:49.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:13:49.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T09:13:49.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T09:13:49.238 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:13:49.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:13:49.239 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T09:13:49.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T09:13:49.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:13:49.240 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:13:49.240 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:13:49.256 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:13:49.254+0000 7f7be78fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:13:49.258 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:13:49.256+0000 7f7be78fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:13:49.259 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:13:49.257+0000 7f7be78fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:13:49.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T09:13:49.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:13:49.617 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:13:49.617 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:13:49.617 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:13:49.617 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:13:49.618 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:13:49.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:13:49.619 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:13:49.619 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:13:49.779 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:13:49.778+0000 7f7be78fd0c0 -1 Falling back to public interface 2021-05-12T09:13:50.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:13:51.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:13:51.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:13:51.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:13:51.023 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:13:51.024 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:13:51.024 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:13:51.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:13:52.303 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:13:52.302+0000 7f7be78fd0c0 -1 osd.0 32 log_to_monitors {default=true} 2021-05-12T09:13:52.446 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:13:52.447 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:13:52.447 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:13:52.449 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:13:52.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:13:52.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:13:52.872 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up out weight 0 up_from 40 up_thru 0 down_at 34 last_clean_interval [5,32) [v2:127.0.0.1:6802/76962,v1:127.0.0.1:6803/76962] [v2:127.0.0.1:6804/76962,v1:127.0.0.1:6805/76962] exists,up 211d7d1b-7ece-4928-932a-3aa488ac19a5 2021-05-12T09:13:52.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:13:52.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:13:52.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:13:52.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2021-05-12T09:13:52.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 1 2021-05-12T09:13:52.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:13:52.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:13:52.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T09:13:52.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:13:52.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T09:13:52.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:13:52.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:13:52.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:13:52.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:13:52.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T09:13:52.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T09:13:52.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:13:52.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:13:52.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:13:52.885 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:13:52.886 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:13:52.886 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:13:52.887 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:13:52.887 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:13:52.887 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:13:52.888 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:13:52.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:13:52.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:13:52.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:13:52.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:13:52.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:13:52.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:13:52.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:13:52.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:13:52.900 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:13:52.900 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T09:13:52.901 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:13:52.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T09:13:52.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:13:52.903 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T09:13:52.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T09:13:52.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:13:52.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:13:52.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:13:52.906 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:13:52.902+0000 7f62b635e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:13:52.906 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:13:52.904+0000 7f62b635e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:13:52.908 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:13:52.905+0000 7f62b635e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:13:53.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T09:13:53.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:13:53.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:13:53.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:13:53.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:13:53.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:13:53.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:13:53.278 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:13:53.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:13:53.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:13:53.423 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:13:53.422+0000 7f62b635e0c0 -1 Falling back to public interface 2021-05-12T09:13:53.655 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:13:54.656 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:13:54.656 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:13:54.656 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:13:54.657 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:13:54.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:13:54.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:13:55.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:13:56.055 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:13:56.055 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:13:56.055 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:13:56.056 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:13:56.056 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:13:56.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:13:56.065 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:13:56.064+0000 7f62b635e0c0 -1 osd.1 32 log_to_monitors {default=true} 2021-05-12T09:13:56.523 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up out weight 0 up_from 42 up_thru 16 down_at 35 last_clean_interval [10,32) [v2:127.0.0.1:6810/77519,v1:127.0.0.1:6811/77519] [v2:127.0.0.1:6812/77519,v1:127.0.0.1:6813/77519] exists,up 860f4a3c-1b2f-41ed-b1b5-8d8eaa20b04b 2021-05-12T09:13:56.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:13:56.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:13:56.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:13:56.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2021-05-12T09:13:56.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2021-05-12T09:13:57.290 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already in. 2021-05-12T09:13:57.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2021-05-12T09:13:57.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.1 2021-05-12T09:13:57.807 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already in. 2021-05-12T09:13:57.817 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2021-05-12T09:14:02.818 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2021-05-12T09:14:02.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2021-05-12T09:14:02.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2021-05-12T09:14:02.820 INFO:tasks.workunit.client.0.smithi028.stdout:writing non-divergent object existing_4 2021-05-12T09:14:02.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T09:14:03.201 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:14:03.201 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-05-12T09:13:58.256161+0000 32'100 46:120 [NONE,1,0] 1 [NONE,1,0] 1 0'0 2021-05-12T09:12:53.510061+0000 0'0 2021-05-12T09:12:53.510061+0000 0 2021-05-12T09:14:03.202 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:13:58.324818+0000 32'62 46:115 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:12:49.062969+0000 0'0 2021-05-12T09:12:49.062969+0000 0 2021-05-12T09:14:03.202 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:14:03.202 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:14:03.203 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:14:03.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2021-05-12T09:14:03.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T09:14:03.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2021-05-12T09:14:03.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:14:03.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:14:03.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:14:03.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:14:03.338 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T09:14:03.338 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:14:03.338 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:14:03.339 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:14:03.339 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:14:03.339 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:14:03.395 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:14:03.395 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:14:03.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:14:03.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:14:03.396 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:14:03.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:14:03.777 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:14:03.777 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:14:03.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:14:03.778 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:14:03.778 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:14:03.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=171798691845 2021-05-12T09:14:03.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 171798691845 2021-05-12T09:14:03.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-171798691845' 2021-05-12T09:14:03.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:14:03.918 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:14:04.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=180388626436 2021-05-12T09:14:04.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 180388626436 2021-05-12T09:14:04.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-171798691845 1-180388626436' 2021-05-12T09:14:04.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:14:04.058 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:14:04.161 INFO:tasks.workunit.client.0.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-05-12T09:14:04.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T09:14:04.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T09:14:04.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T09:14:04.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:14:04.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-171798691845 2021-05-12T09:14:04.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:14:04.164 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:14:04.165 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-171798691845 2021-05-12T09:14:04.166 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:14:04.166 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=171798691845 2021-05-12T09:14:04.166 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2021-05-12T09:14:04.166 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 171798691845 2021-05-12T09:14:04.167 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:14:04.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 171798691843 -lt 171798691845 2021-05-12T09:14:04.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:14:05.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:14:05.654 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:14:06.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 171798691845 -lt 171798691845 2021-05-12T09:14:06.049 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:14:06.050 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-180388626436 2021-05-12T09:14:06.050 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:14:06.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:14:06.051 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-180388626436 2021-05-12T09:14:06.051 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:14:06.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=180388626436 2021-05-12T09:14:06.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 180388626436' 2021-05-12T09:14:06.053 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 180388626436 2021-05-12T09:14:06.053 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:14:06.464 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 180388626436 -lt 180388626436 2021-05-12T09:14:06.465 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:14:06.465 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:06.465 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:06.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:14:06.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:14:06.970 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:14:06.970 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:14:06.970 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:14:06.970 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:14:06.971 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:14:06.971 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:14:07.352 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:14:07.353 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:14:07.354 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:07.354 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:07.854 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:14:07.854 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T09:14:07.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T09:14:07.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T09:14:07.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:14:07.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:14:07.955 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:14:07.956 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:14:07.956 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:14:07.956 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:14:07.957 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:14:07.957 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:14:07.957 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:14:07.957 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:14:08.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:14:08.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:14:08.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:08.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:08.843 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:14:08.843 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:14:08.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:14:08.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:14:08.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:14:08.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:14:08.845 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:14:08.845 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:14:08.845 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:14:09.347 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:14:09.347 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:14:09.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T09:14:09.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:14:09.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T09:14:09.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:14:09.549 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:14:09.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:14:09.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:14:09.550 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:14:09.550 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:14:09.550 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:14:09.550 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:14:09.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:14:09.928 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:14:09.929 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:09.929 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:10.434 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:14:10.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:14:10.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:14:10.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:14:10.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:14:10.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:14:10.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:14:10.495 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:14:10.495 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:14:10.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:14:10.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:14:10.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T09:14:10.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:14:10.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T09:14:11.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:14:11.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:14:11.345 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:14:11.345 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:14:11.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:14:11.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:14:11.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:14:11.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:14:11.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:14:11.729 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:14:11.729 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:11.729 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:12.242 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:14:12.243 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:14:12.243 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:14:12.243 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:14:12.243 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:14:12.243 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:14:12.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:14:12.244 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:14:12.244 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:14:12.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:14:12.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:14:12.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T09:14:12.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:14:12.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T09:14:13.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:14:13.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:14:13.548 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:14:13.548 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:14:13.548 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:14:13.548 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:14:13.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:14:13.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:14:13.933 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:14:13.933 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:14:13.934 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:13.934 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:14.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:14:14.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:14:14.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:14:14.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:14:14.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:14:14.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:14:14.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:14:14.441 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:14:14.441 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:14:14.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:14:14.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:14:14.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T09:14:14.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:14:14.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T09:14:16.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:14:16.549 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:14:16.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:14:16.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:14:16.550 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:14:16.550 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:14:16.550 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:14:16.550 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:14:16.933 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:14:16.934 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:14:16.934 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:16.934 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:17.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:14:17.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:14:17.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:14:17.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:14:17.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:14:17.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:14:17.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:14:17.440 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:14:17.440 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:14:17.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:14:17.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:14:17.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T09:14:17.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:14:17.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T09:14:21.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:14:21.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:14:21.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:14:21.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:14:21.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:14:21.146 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:14:21.146 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:14:21.146 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:14:21.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:14:21.536 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:14:21.536 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:21.536 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:22.042 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:14:22.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:14:22.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:14:22.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:14:22.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:14:22.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:14:22.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:14:22.044 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:14:22.044 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:14:22.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:14:22.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:14:22.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T09:14:22.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:14:22.566 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T09:14:28.966 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:14:28.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:14:28.967 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:14:28.967 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:14:28.968 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:14:28.968 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:14:28.968 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:14:28.968 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:14:29.353 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:14:29.353 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:14:29.354 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:29.354 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:29.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:14:29.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:14:29.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:14:29.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:14:29.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:14:29.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:14:29.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:14:29.871 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:14:29.872 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:14:30.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:14:30.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:14:30.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T09:14:30.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:14:30.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T09:14:37.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:14:37.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:14:37.683 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:14:37.683 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:14:37.683 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:14:37.683 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:14:37.684 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:14:37.684 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:14:38.073 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:14:38.073 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:14:38.074 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:38.074 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:38.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:14:38.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:14:38.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:14:38.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:14:38.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:14:38.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:14:38.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:14:38.585 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:14:38.585 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:14:39.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:14:39.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:14:39.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T09:14:39.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T09:14:39.585 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:14:39.585 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_fingerprint": "29208aba-34ab-4a89-ac1b-0d5e6428bb6d", 2021-05-12T09:14:39.586 INFO:tasks.workunit.client.0.smithi028.stdout: "version": "17.0.0-4095-ga8c673d5", 2021-05-12T09:14:39.586 INFO:tasks.workunit.client.0.smithi028.stdout: "commit": "a8c673d555488beec9ccda5812ba822e69c12ea9", 2021-05-12T09:14:39.586 INFO:tasks.workunit.client.0.smithi028.stdout: "timestamp": "2021-05-12T09:14:39.581293+0000", 2021-05-12T09:14:39.586 INFO:tasks.workunit.client.0.smithi028.stdout: "tag": "", 2021-05-12T09:14:39.586 INFO:tasks.workunit.client.0.smithi028.stdout: "health": { 2021-05-12T09:14:39.587 INFO:tasks.workunit.client.0.smithi028.stdout: "status": "HEALTH_WARN", 2021-05-12T09:14:39.587 INFO:tasks.workunit.client.0.smithi028.stdout: "checks": { 2021-05-12T09:14:39.587 INFO:tasks.workunit.client.0.smithi028.stdout: "OSDMAP_FLAGS": { 2021-05-12T09:14:39.587 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:14:39.588 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:14:39.588 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T09:14:39.588 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 17 2021-05-12T09:14:39.588 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.589 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [], 2021-05-12T09:14:39.589 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:14:39.589 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.589 INFO:tasks.workunit.client.0.smithi028.stdout: "OSD_DOWN": { 2021-05-12T09:14:39.589 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:14:39.590 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:14:39.590 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "1 osds down", 2021-05-12T09:14:39.590 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 1 2021-05-12T09:14:39.590 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.591 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:14:39.591 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.591 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "osd.2 (root=default,host=smithi028) is down" 2021-05-12T09:14:39.591 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.591 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.592 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:14:39.592 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.592 INFO:tasks.workunit.client.0.smithi028.stdout: "PG_DEGRADED": { 2021-05-12T09:14:39.592 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:14:39.592 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:14:39.593 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "Degraded data redundancy: 103/309 objects degraded (33.333%), 2 pgs degraded", 2021-05-12T09:14:39.593 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 2 2021-05-12T09:14:39.593 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.593 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:14:39.594 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.594 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:14:39.594 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.594 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.594 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.0 is active+undersized+degraded, acting [NONE,1,0]" 2021-05-12T09:14:39.595 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.595 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.595 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:14:39.595 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.596 INFO:tasks.workunit.client.0.smithi028.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T09:14:39.596 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:14:39.596 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:14:39.596 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T09:14:39.596 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 1 2021-05-12T09:14:39.597 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.597 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:14:39.597 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.597 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T09:14:39.598 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.598 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.598 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T09:14:39.598 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.599 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.599 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:14:39.599 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.599 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.599 INFO:tasks.workunit.client.0.smithi028.stdout: "mutes": [] 2021-05-12T09:14:39.600 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.600 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap_first_committed": 1, 2021-05-12T09:14:39.600 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap_last_committed": 1, 2021-05-12T09:14:39.600 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap": { 2021-05-12T09:14:39.601 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 1, 2021-05-12T09:14:39.601 INFO:tasks.workunit.client.0.smithi028.stdout: "fsid": "8342ec9e-531f-44b4-8c8c-079d667fe908", 2021-05-12T09:14:39.601 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:12:20.498549Z", 2021-05-12T09:14:39.601 INFO:tasks.workunit.client.0.smithi028.stdout: "created": "2021-05-12T09:12:20.498549Z", 2021-05-12T09:14:39.601 INFO:tasks.workunit.client.0.smithi028.stdout: "min_mon_release": 17, 2021-05-12T09:14:39.602 INFO:tasks.workunit.client.0.smithi028.stdout: "min_mon_release_name": "quincy", 2021-05-12T09:14:39.602 INFO:tasks.workunit.client.0.smithi028.stdout: "election_strategy": 1, 2021-05-12T09:14:39.602 INFO:tasks.workunit.client.0.smithi028.stdout: "disallowed_leaders: ": "", 2021-05-12T09:14:39.602 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode": false, 2021-05-12T09:14:39.603 INFO:tasks.workunit.client.0.smithi028.stdout: "features": { 2021-05-12T09:14:39.603 INFO:tasks.workunit.client.0.smithi028.stdout: "persistent": [ 2021-05-12T09:14:39.603 INFO:tasks.workunit.client.0.smithi028.stdout: "kraken", 2021-05-12T09:14:39.603 INFO:tasks.workunit.client.0.smithi028.stdout: "luminous", 2021-05-12T09:14:39.604 INFO:tasks.workunit.client.0.smithi028.stdout: "mimic", 2021-05-12T09:14:39.604 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap-prune", 2021-05-12T09:14:39.604 INFO:tasks.workunit.client.0.smithi028.stdout: "nautilus", 2021-05-12T09:14:39.604 INFO:tasks.workunit.client.0.smithi028.stdout: "octopus", 2021-05-12T09:14:39.604 INFO:tasks.workunit.client.0.smithi028.stdout: "pacific", 2021-05-12T09:14:39.605 INFO:tasks.workunit.client.0.smithi028.stdout: "elector-pinging", 2021-05-12T09:14:39.605 INFO:tasks.workunit.client.0.smithi028.stdout: "quincy" 2021-05-12T09:14:39.605 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.605 INFO:tasks.workunit.client.0.smithi028.stdout: "optional": [] 2021-05-12T09:14:39.606 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.606 INFO:tasks.workunit.client.0.smithi028.stdout: "mons": [ 2021-05-12T09:14:39.606 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.606 INFO:tasks.workunit.client.0.smithi028.stdout: "rank": 0, 2021-05-12T09:14:39.607 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "a", 2021-05-12T09:14:39.607 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:14:39.607 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:14:39.607 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.607 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:14:39.608 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:7115", 2021-05-12T09:14:39.608 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 0 2021-05-12T09:14:39.608 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.608 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.608 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.609 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T09:14:39.609 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T09:14:39.609 INFO:tasks.workunit.client.0.smithi028.stdout: "priority": 0, 2021-05-12T09:14:39.609 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 0, 2021-05-12T09:14:39.610 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_location": "{}" 2021-05-12T09:14:39.610 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.610 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.610 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.610 INFO:tasks.workunit.client.0.smithi028.stdout: "quorum": [ 2021-05-12T09:14:39.611 INFO:tasks.workunit.client.0.smithi028.stdout: 0 2021-05-12T09:14:39.611 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.611 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap": { 2021-05-12T09:14:39.611 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 46, 2021-05-12T09:14:39.611 INFO:tasks.workunit.client.0.smithi028.stdout: "fsid": "8342ec9e-531f-44b4-8c8c-079d667fe908", 2021-05-12T09:14:39.612 INFO:tasks.workunit.client.0.smithi028.stdout: "created": "2021-05-12T09:12:21.349859+0000", 2021-05-12T09:14:39.612 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:13:57.840031+0000", 2021-05-12T09:14:39.612 INFO:tasks.workunit.client.0.smithi028.stdout: "last_up_change": "2021-05-12T09:13:56.368970+0000", 2021-05-12T09:14:39.612 INFO:tasks.workunit.client.0.smithi028.stdout: "last_in_change": "2021-05-12T09:13:57.628069+0000", 2021-05-12T09:14:39.613 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T09:14:39.613 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_num": 5800384, 2021-05-12T09:14:39.613 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_set": [ 2021-05-12T09:14:39.613 INFO:tasks.workunit.client.0.smithi028.stdout: "nodown", 2021-05-12T09:14:39.613 INFO:tasks.workunit.client.0.smithi028.stdout: "noin", 2021-05-12T09:14:39.614 INFO:tasks.workunit.client.0.smithi028.stdout: "noout", 2021-05-12T09:14:39.614 INFO:tasks.workunit.client.0.smithi028.stdout: "pglog_hardlimit", 2021-05-12T09:14:39.614 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snapdirs", 2021-05-12T09:14:39.614 INFO:tasks.workunit.client.0.smithi028.stdout: "recovery_deletes", 2021-05-12T09:14:39.614 INFO:tasks.workunit.client.0.smithi028.stdout: "sortbitwise" 2021-05-12T09:14:39.615 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.615 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_version": 10, 2021-05-12T09:14:39.615 INFO:tasks.workunit.client.0.smithi028.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T09:14:39.615 INFO:tasks.workunit.client.0.smithi028.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T09:14:39.616 INFO:tasks.workunit.client.0.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T09:14:39.616 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_snapshot": "", 2021-05-12T09:14:39.616 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_max": 2, 2021-05-12T09:14:39.616 INFO:tasks.workunit.client.0.smithi028.stdout: "max_osd": 3, 2021-05-12T09:14:39.616 INFO:tasks.workunit.client.0.smithi028.stdout: "require_min_compat_client": "luminous", 2021-05-12T09:14:39.617 INFO:tasks.workunit.client.0.smithi028.stdout: "min_compat_client": "jewel", 2021-05-12T09:14:39.617 INFO:tasks.workunit.client.0.smithi028.stdout: "require_osd_release": "quincy", 2021-05-12T09:14:39.617 INFO:tasks.workunit.client.0.smithi028.stdout: "pools": [ 2021-05-12T09:14:39.617 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.617 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 1, 2021-05-12T09:14:39.618 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_name": ".mgr", 2021-05-12T09:14:39.618 INFO:tasks.workunit.client.0.smithi028.stdout: "create_time": "2021-05-12T09:12:49.004998+0000", 2021-05-12T09:14:39.618 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": 1, 2021-05-12T09:14:39.618 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_names": "hashpspool", 2021-05-12T09:14:39.619 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:14:39.619 INFO:tasks.workunit.client.0.smithi028.stdout: "size": 3, 2021-05-12T09:14:39.619 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 2, 2021-05-12T09:14:39.619 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_rule": 0, 2021-05-12T09:14:39.619 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_count": 0, 2021-05-12T09:14:39.620 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_target": 0, 2021-05-12T09:14:39.620 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T09:14:39.620 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T09:14:39.620 INFO:tasks.workunit.client.0.smithi028.stdout: "object_hash": 2, 2021-05-12T09:14:39.621 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_autoscale_mode": "off", 2021-05-12T09:14:39.621 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num": 1, 2021-05-12T09:14:39.621 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num": 1, 2021-05-12T09:14:39.621 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num_target": 1, 2021-05-12T09:14:39.621 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_target": 1, 2021-05-12T09:14:39.622 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_pending": 1, 2021-05-12T09:14:39.622 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pg_merge_meta": { 2021-05-12T09:14:39.622 INFO:tasks.workunit.client.0.smithi028.stdout: "source_pgid": "0.0", 2021-05-12T09:14:39.622 INFO:tasks.workunit.client.0.smithi028.stdout: "ready_epoch": 0, 2021-05-12T09:14:39.622 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_started": 0, 2021-05-12T09:14:39.623 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": 0, 2021-05-12T09:14:39.623 INFO:tasks.workunit.client.0.smithi028.stdout: "source_version": "0'0", 2021-05-12T09:14:39.623 INFO:tasks.workunit.client.0.smithi028.stdout: "target_version": "0'0" 2021-05-12T09:14:39.623 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.624 INFO:tasks.workunit.client.0.smithi028.stdout: "last_change": "21", 2021-05-12T09:14:39.624 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend": "0", 2021-05-12T09:14:39.624 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T09:14:39.624 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T09:14:39.624 INFO:tasks.workunit.client.0.smithi028.stdout: "auid": 0, 2021-05-12T09:14:39.625 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_mode": "selfmanaged", 2021-05-12T09:14:39.625 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_seq": 0, 2021-05-12T09:14:39.625 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_epoch": 0, 2021-05-12T09:14:39.625 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_snaps": [], 2021-05-12T09:14:39.625 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps": "[]", 2021-05-12T09:14:39.626 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_bytes": 0, 2021-05-12T09:14:39.626 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_objects": 0, 2021-05-12T09:14:39.626 INFO:tasks.workunit.client.0.smithi028.stdout: "tiers": [], 2021-05-12T09:14:39.626 INFO:tasks.workunit.client.0.smithi028.stdout: "tier_of": -1, 2021-05-12T09:14:39.627 INFO:tasks.workunit.client.0.smithi028.stdout: "read_tier": -1, 2021-05-12T09:14:39.627 INFO:tasks.workunit.client.0.smithi028.stdout: "write_tier": -1, 2021-05-12T09:14:39.627 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_mode": "none", 2021-05-12T09:14:39.627 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_bytes": 0, 2021-05-12T09:14:39.627 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_objects": 0, 2021-05-12T09:14:39.628 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T09:14:39.628 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T09:14:39.628 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T09:14:39.628 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_flush_age": 0, 2021-05-12T09:14:39.628 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_evict_age": 0, 2021-05-12T09:14:39.629 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profile": "", 2021-05-12T09:14:39.629 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_params": { 2021-05-12T09:14:39.629 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "none" 2021-05-12T09:14:39.629 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.629 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_period": 0, 2021-05-12T09:14:39.630 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_count": 0, 2021-05-12T09:14:39.630 INFO:tasks.workunit.client.0.smithi028.stdout: "use_gmt_hitset": true, 2021-05-12T09:14:39.630 INFO:tasks.workunit.client.0.smithi028.stdout: "min_read_recency_for_promote": 0, 2021-05-12T09:14:39.630 INFO:tasks.workunit.client.0.smithi028.stdout: "min_write_recency_for_promote": 0, 2021-05-12T09:14:39.631 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T09:14:39.631 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_search_last_n": 0, 2021-05-12T09:14:39.631 INFO:tasks.workunit.client.0.smithi028.stdout: "grade_table": [], 2021-05-12T09:14:39.631 INFO:tasks.workunit.client.0.smithi028.stdout: "stripe_width": 0, 2021-05-12T09:14:39.631 INFO:tasks.workunit.client.0.smithi028.stdout: "expected_num_objects": 0, 2021-05-12T09:14:39.632 INFO:tasks.workunit.client.0.smithi028.stdout: "fast_read": false, 2021-05-12T09:14:39.632 INFO:tasks.workunit.client.0.smithi028.stdout: "options": { 2021-05-12T09:14:39.632 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_min": 1 2021-05-12T09:14:39.632 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.632 INFO:tasks.workunit.client.0.smithi028.stdout: "application_metadata": { 2021-05-12T09:14:39.633 INFO:tasks.workunit.client.0.smithi028.stdout: "mgr": {} 2021-05-12T09:14:39.633 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.633 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.633 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.634 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 2, 2021-05-12T09:14:39.634 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_name": "test", 2021-05-12T09:14:39.634 INFO:tasks.workunit.client.0.smithi028.stdout: "create_time": "2021-05-12T09:12:53.464789+0000", 2021-05-12T09:14:39.634 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": 16389, 2021-05-12T09:14:39.634 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2021-05-12T09:14:39.635 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 3, 2021-05-12T09:14:39.635 INFO:tasks.workunit.client.0.smithi028.stdout: "size": 3, 2021-05-12T09:14:39.635 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 2, 2021-05-12T09:14:39.635 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_rule": 1, 2021-05-12T09:14:39.636 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_count": 0, 2021-05-12T09:14:39.636 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_target": 0, 2021-05-12T09:14:39.636 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T09:14:39.636 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T09:14:39.636 INFO:tasks.workunit.client.0.smithi028.stdout: "object_hash": 2, 2021-05-12T09:14:39.637 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_autoscale_mode": "off", 2021-05-12T09:14:39.637 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num": 1, 2021-05-12T09:14:39.637 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num": 1, 2021-05-12T09:14:39.637 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num_target": 1, 2021-05-12T09:14:39.637 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_target": 1, 2021-05-12T09:14:39.638 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_pending": 1, 2021-05-12T09:14:39.638 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pg_merge_meta": { 2021-05-12T09:14:39.638 INFO:tasks.workunit.client.0.smithi028.stdout: "source_pgid": "0.0", 2021-05-12T09:14:39.638 INFO:tasks.workunit.client.0.smithi028.stdout: "ready_epoch": 0, 2021-05-12T09:14:39.639 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_started": 0, 2021-05-12T09:14:39.639 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": 0, 2021-05-12T09:14:39.639 INFO:tasks.workunit.client.0.smithi028.stdout: "source_version": "0'0", 2021-05-12T09:14:39.639 INFO:tasks.workunit.client.0.smithi028.stdout: "target_version": "0'0" 2021-05-12T09:14:39.639 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.640 INFO:tasks.workunit.client.0.smithi028.stdout: "last_change": "33", 2021-05-12T09:14:39.640 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend": "0", 2021-05-12T09:14:39.640 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T09:14:39.640 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T09:14:39.641 INFO:tasks.workunit.client.0.smithi028.stdout: "auid": 0, 2021-05-12T09:14:39.641 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_mode": "pool", 2021-05-12T09:14:39.641 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_seq": 1, 2021-05-12T09:14:39.641 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_epoch": 33, 2021-05-12T09:14:39.641 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_snaps": [ 2021-05-12T09:14:39.642 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.642 INFO:tasks.workunit.client.0.smithi028.stdout: "snapid": 1, 2021-05-12T09:14:39.642 INFO:tasks.workunit.client.0.smithi028.stdout: "stamp": "2021-05-12T09:13:35.394733+0000", 2021-05-12T09:14:39.642 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "snap1" 2021-05-12T09:14:39.642 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.643 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.643 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps": "[]", 2021-05-12T09:14:39.643 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_bytes": 0, 2021-05-12T09:14:39.643 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_objects": 0, 2021-05-12T09:14:39.644 INFO:tasks.workunit.client.0.smithi028.stdout: "tiers": [], 2021-05-12T09:14:39.644 INFO:tasks.workunit.client.0.smithi028.stdout: "tier_of": -1, 2021-05-12T09:14:39.644 INFO:tasks.workunit.client.0.smithi028.stdout: "read_tier": -1, 2021-05-12T09:14:39.644 INFO:tasks.workunit.client.0.smithi028.stdout: "write_tier": -1, 2021-05-12T09:14:39.644 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_mode": "none", 2021-05-12T09:14:39.645 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_bytes": 0, 2021-05-12T09:14:39.645 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_objects": 0, 2021-05-12T09:14:39.645 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T09:14:39.645 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T09:14:39.646 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T09:14:39.646 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_flush_age": 0, 2021-05-12T09:14:39.646 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_evict_age": 0, 2021-05-12T09:14:39.646 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profile": "myprofile", 2021-05-12T09:14:39.646 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_params": { 2021-05-12T09:14:39.647 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "none" 2021-05-12T09:14:39.647 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.647 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_period": 0, 2021-05-12T09:14:39.647 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_count": 0, 2021-05-12T09:14:39.647 INFO:tasks.workunit.client.0.smithi028.stdout: "use_gmt_hitset": true, 2021-05-12T09:14:39.648 INFO:tasks.workunit.client.0.smithi028.stdout: "min_read_recency_for_promote": 0, 2021-05-12T09:14:39.648 INFO:tasks.workunit.client.0.smithi028.stdout: "min_write_recency_for_promote": 0, 2021-05-12T09:14:39.648 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T09:14:39.648 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_search_last_n": 0, 2021-05-12T09:14:39.649 INFO:tasks.workunit.client.0.smithi028.stdout: "grade_table": [], 2021-05-12T09:14:39.649 INFO:tasks.workunit.client.0.smithi028.stdout: "stripe_width": 8192, 2021-05-12T09:14:39.649 INFO:tasks.workunit.client.0.smithi028.stdout: "expected_num_objects": 0, 2021-05-12T09:14:39.649 INFO:tasks.workunit.client.0.smithi028.stdout: "fast_read": false, 2021-05-12T09:14:39.649 INFO:tasks.workunit.client.0.smithi028.stdout: "options": {}, 2021-05-12T09:14:39.650 INFO:tasks.workunit.client.0.smithi028.stdout: "application_metadata": {} 2021-05-12T09:14:39.650 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.650 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.650 INFO:tasks.workunit.client.0.smithi028.stdout: "osds": [ 2021-05-12T09:14:39.650 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.651 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:14:39.651 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "211d7d1b-7ece-4928-932a-3aa488ac19a5", 2021-05-12T09:14:39.651 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 1, 2021-05-12T09:14:39.651 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:14:39.652 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:14:39.652 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:14:39.652 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 5, 2021-05-12T09:14:39.652 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 32, 2021-05-12T09:14:39.652 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 40, 2021-05-12T09:14:39.653 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 43, 2021-05-12T09:14:39.653 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 34, 2021-05-12T09:14:39.653 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:14:39.653 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:14:39.653 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:14:39.654 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.654 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:14:39.654 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6802", 2021-05-12T09:14:39.654 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 76962 2021-05-12T09:14:39.654 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.655 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.655 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:14:39.655 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6803", 2021-05-12T09:14:39.655 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 76962 2021-05-12T09:14:39.656 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.656 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.656 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.656 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:14:39.656 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:14:39.657 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.657 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:14:39.657 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6804", 2021-05-12T09:14:39.657 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 76962 2021-05-12T09:14:39.657 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.658 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.658 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:14:39.658 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6805", 2021-05-12T09:14:39.658 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 76962 2021-05-12T09:14:39.659 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.659 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.659 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.659 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:14:39.659 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:14:39.660 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.660 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:14:39.660 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6808", 2021-05-12T09:14:39.660 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 76962 2021-05-12T09:14:39.660 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.661 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.661 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:14:39.661 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6809", 2021-05-12T09:14:39.661 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 76962 2021-05-12T09:14:39.661 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.662 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.662 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.662 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:14:39.662 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:14:39.663 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.663 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:14:39.663 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6806", 2021-05-12T09:14:39.663 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 76962 2021-05-12T09:14:39.663 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.664 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.664 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:14:39.664 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6807", 2021-05-12T09:14:39.664 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 76962 2021-05-12T09:14:39.664 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.665 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.665 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.665 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6803/76962", 2021-05-12T09:14:39.665 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6805/76962", 2021-05-12T09:14:39.666 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6809/76962", 2021-05-12T09:14:39.666 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6807/76962", 2021-05-12T09:14:39.666 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:14:39.666 INFO:tasks.workunit.client.0.smithi028.stdout: "exists", 2021-05-12T09:14:39.666 INFO:tasks.workunit.client.0.smithi028.stdout: "up" 2021-05-12T09:14:39.667 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.667 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.667 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.667 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:14:39.667 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "860f4a3c-1b2f-41ed-b1b5-8d8eaa20b04b", 2021-05-12T09:14:39.668 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 1, 2021-05-12T09:14:39.668 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:14:39.668 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:14:39.668 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:14:39.669 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 10, 2021-05-12T09:14:39.669 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 32, 2021-05-12T09:14:39.669 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 42, 2021-05-12T09:14:39.669 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 45, 2021-05-12T09:14:39.669 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 35, 2021-05-12T09:14:39.670 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:14:39.670 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:14:39.670 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:14:39.670 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.670 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:14:39.671 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6810", 2021-05-12T09:14:39.671 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 77519 2021-05-12T09:14:39.671 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.671 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.671 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:14:39.672 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6811", 2021-05-12T09:14:39.672 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 77519 2021-05-12T09:14:39.672 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.672 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.673 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.673 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:14:39.673 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:14:39.673 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.673 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:14:39.674 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6812", 2021-05-12T09:14:39.674 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 77519 2021-05-12T09:14:39.674 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.674 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.674 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:14:39.675 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6813", 2021-05-12T09:14:39.675 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 77519 2021-05-12T09:14:39.675 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.675 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.676 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.676 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:14:39.676 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:14:39.676 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.676 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:14:39.677 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6816", 2021-05-12T09:14:39.677 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 77519 2021-05-12T09:14:39.677 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.677 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.678 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:14:39.678 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6817", 2021-05-12T09:14:39.678 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 77519 2021-05-12T09:14:39.678 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.678 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.679 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.679 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:14:39.679 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:14:39.679 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.679 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:14:39.680 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6814", 2021-05-12T09:14:39.680 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 77519 2021-05-12T09:14:39.680 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.680 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.680 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:14:39.681 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6815", 2021-05-12T09:14:39.681 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 77519 2021-05-12T09:14:39.681 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.681 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.682 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.682 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6811/77519", 2021-05-12T09:14:39.682 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6813/77519", 2021-05-12T09:14:39.682 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6817/77519", 2021-05-12T09:14:39.682 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6815/77519", 2021-05-12T09:14:39.683 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:14:39.683 INFO:tasks.workunit.client.0.smithi028.stdout: "exists", 2021-05-12T09:14:39.683 INFO:tasks.workunit.client.0.smithi028.stdout: "up" 2021-05-12T09:14:39.683 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.683 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.684 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.684 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:14:39.684 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "7951a78e-ee8e-4c4f-a30b-05cc781094fa", 2021-05-12T09:14:39.685 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 0, 2021-05-12T09:14:39.685 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:14:39.685 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:14:39.685 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:14:39.685 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 0, 2021-05-12T09:14:39.686 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 0, 2021-05-12T09:14:39.686 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 15, 2021-05-12T09:14:39.686 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 28, 2021-05-12T09:14:39.686 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 36, 2021-05-12T09:14:39.687 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:14:39.687 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:14:39.687 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:14:39.687 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.687 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:14:39.688 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6818", 2021-05-12T09:14:39.688 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:14:39.688 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.688 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.688 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:14:39.689 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6819", 2021-05-12T09:14:39.689 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:14:39.689 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.689 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.690 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.690 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:14:39.690 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:14:39.690 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.690 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:14:39.691 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6820", 2021-05-12T09:14:39.691 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:14:39.691 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.691 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.692 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:14:39.692 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6821", 2021-05-12T09:14:39.692 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:14:39.692 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.692 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.693 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.693 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:14:39.693 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:14:39.693 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.693 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:14:39.694 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6824", 2021-05-12T09:14:39.694 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:14:39.694 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.694 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.694 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:14:39.695 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6825", 2021-05-12T09:14:39.695 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:14:39.695 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.695 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.696 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.696 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:14:39.696 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:14:39.696 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.696 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:14:39.697 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6822", 2021-05-12T09:14:39.697 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:14:39.697 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.697 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.698 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:14:39.698 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6823", 2021-05-12T09:14:39.698 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:14:39.698 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.698 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.699 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.699 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6819/72533", 2021-05-12T09:14:39.699 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6821/72533", 2021-05-12T09:14:39.699 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6825/72533", 2021-05-12T09:14:39.699 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6823/72533", 2021-05-12T09:14:39.700 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:14:39.700 INFO:tasks.workunit.client.0.smithi028.stdout: "exists" 2021-05-12T09:14:39.700 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.700 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.701 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.701 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_xinfo": [ 2021-05-12T09:14:39.701 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.701 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:14:39.701 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:13:46.392119+0000", 2021-05-12T09:14:39.702 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:14:39.702 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:14:39.702 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:14:39.702 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:14:39.703 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:12:30.963646+0000", 2021-05-12T09:14:39.703 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:14:39.703 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.703 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.703 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:14:39.704 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:13:46.970848+0000", 2021-05-12T09:14:39.704 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:14:39.704 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:14:39.704 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:14:39.705 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:14:39.705 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:12:40.136304+0000", 2021-05-12T09:14:39.705 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:14:39.705 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.705 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.706 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:14:39.706 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:13:47.536721+0000", 2021-05-12T09:14:39.706 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:14:39.706 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:14:39.707 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:14:39.707 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:14:39.707 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:12:49.387397+0000", 2021-05-12T09:14:39.707 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:14:39.707 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.708 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.708 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_upmap": [], 2021-05-12T09:14:39.708 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_upmap_items": [], 2021-05-12T09:14:39.708 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_temp": [], 2021-05-12T09:14:39.709 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_temp": [], 2021-05-12T09:14:39.709 INFO:tasks.workunit.client.0.smithi028.stdout: "blocklist": {}, 2021-05-12T09:14:39.709 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profiles": { 2021-05-12T09:14:39.709 INFO:tasks.workunit.client.0.smithi028.stdout: "default": { 2021-05-12T09:14:39.709 INFO:tasks.workunit.client.0.smithi028.stdout: "k": "2", 2021-05-12T09:14:39.710 INFO:tasks.workunit.client.0.smithi028.stdout: "m": "2", 2021-05-12T09:14:39.710 INFO:tasks.workunit.client.0.smithi028.stdout: "plugin": "jerasure", 2021-05-12T09:14:39.710 INFO:tasks.workunit.client.0.smithi028.stdout: "technique": "reed_sol_van" 2021-05-12T09:14:39.710 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.711 INFO:tasks.workunit.client.0.smithi028.stdout: "myprofile": { 2021-05-12T09:14:39.711 INFO:tasks.workunit.client.0.smithi028.stdout: "crush-device-class": "", 2021-05-12T09:14:39.711 INFO:tasks.workunit.client.0.smithi028.stdout: "crush-failure-domain": "osd", 2021-05-12T09:14:39.711 INFO:tasks.workunit.client.0.smithi028.stdout: "crush-root": "default", 2021-05-12T09:14:39.711 INFO:tasks.workunit.client.0.smithi028.stdout: "jerasure-per-chunk-alignment": "false", 2021-05-12T09:14:39.712 INFO:tasks.workunit.client.0.smithi028.stdout: "k": "2", 2021-05-12T09:14:39.712 INFO:tasks.workunit.client.0.smithi028.stdout: "m": "1", 2021-05-12T09:14:39.712 INFO:tasks.workunit.client.0.smithi028.stdout: "plugin": "jerasure", 2021-05-12T09:14:39.712 INFO:tasks.workunit.client.0.smithi028.stdout: "technique": "reed_sol_van", 2021-05-12T09:14:39.713 INFO:tasks.workunit.client.0.smithi028.stdout: "w": "8" 2021-05-12T09:14:39.713 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.713 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.713 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps_queue": [], 2021-05-12T09:14:39.713 INFO:tasks.workunit.client.0.smithi028.stdout: "new_removed_snaps": [], 2021-05-12T09:14:39.714 INFO:tasks.workunit.client.0.smithi028.stdout: "new_purged_snaps": [], 2021-05-12T09:14:39.714 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_node_flags": {}, 2021-05-12T09:14:39.714 INFO:tasks.workunit.client.0.smithi028.stdout: "device_class_flags": {}, 2021-05-12T09:14:39.714 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode": { 2021-05-12T09:14:39.715 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode_enabled": false, 2021-05-12T09:14:39.715 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_bucket_count": 0, 2021-05-12T09:14:39.715 INFO:tasks.workunit.client.0.smithi028.stdout: "degraded_stretch_mode": 0, 2021-05-12T09:14:39.715 INFO:tasks.workunit.client.0.smithi028.stdout: "recovering_stretch_mode": 0, 2021-05-12T09:14:39.715 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode_bucket": 0 2021-05-12T09:14:39.716 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.716 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.716 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_metadata": [ 2021-05-12T09:14:39.716 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.717 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:14:39.717 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:14:39.717 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6804/76962,v1:127.0.0.1:6805/76962]", 2021-05-12T09:14:39.717 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:14:39.717 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:14:39.718 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:14:39.718 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:14:39.718 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:14:39.718 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:14:39.719 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:14:39.719 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:14:39.719 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:14:39.719 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T09:14:39.719 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:14:39.720 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:14:39.720 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:14:39.720 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:14:39.720 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:14:39.720 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:14:39.721 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:14:39.721 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:14:39.721 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:14:39.721 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:14:39.722 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:14:39.722 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:14:39.722 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:14:39.722 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:14:39.722 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:14:39.723 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6802/76962,v1:127.0.0.1:6803/76962]", 2021-05-12T09:14:39.723 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:14:39.723 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/76962,v1:127.0.0.1:6809/76962]", 2021-05-12T09:14:39.723 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/76962,v1:127.0.0.1:6807/76962]", 2021-05-12T09:14:39.724 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:14:39.724 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:14:39.724 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:14:39.724 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:14:39.724 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:14:39.725 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:14:39.725 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:14:39.725 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:14:39.725 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:14:39.726 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T09:14:39.726 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:14:39.726 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:14:39.726 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:14:39.726 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.727 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.727 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:14:39.727 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:14:39.727 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6812/77519,v1:127.0.0.1:6813/77519]", 2021-05-12T09:14:39.728 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:14:39.728 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:14:39.728 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:14:39.728 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:14:39.728 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:14:39.729 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:14:39.729 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:14:39.729 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:14:39.729 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:14:39.730 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T09:14:39.730 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:14:39.730 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:14:39.730 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:14:39.730 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:14:39.731 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:14:39.731 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:14:39.731 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:14:39.731 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:14:39.732 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:14:39.732 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:14:39.732 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:14:39.732 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:14:39.733 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:14:39.733 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:14:39.733 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:14:39.733 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6810/77519,v1:127.0.0.1:6811/77519]", 2021-05-12T09:14:39.733 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:14:39.734 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/77519,v1:127.0.0.1:6817/77519]", 2021-05-12T09:14:39.734 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/77519,v1:127.0.0.1:6815/77519]", 2021-05-12T09:14:39.734 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:14:39.734 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:14:39.735 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:14:39.735 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:14:39.735 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:14:39.735 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:14:39.735 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:14:39.736 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:14:39.736 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:14:39.736 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T09:14:39.736 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:14:39.736 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:14:39.737 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:14:39.737 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.737 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.737 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:14:39.738 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:14:39.738 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6820/72533,v1:127.0.0.1:6821/72533]", 2021-05-12T09:14:39.738 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:14:39.738 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:14:39.738 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:14:39.739 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:14:39.739 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:14:39.739 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:14:39.739 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:14:39.740 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:14:39.740 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:14:39.740 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T09:14:39.740 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:14:39.740 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:14:39.741 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:14:39.741 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:14:39.741 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:14:39.741 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:14:39.741 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:14:39.742 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:14:39.742 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:14:39.742 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:14:39.742 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:14:39.743 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:14:39.743 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:14:39.743 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:14:39.743 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:14:39.743 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6818/72533,v1:127.0.0.1:6819/72533]", 2021-05-12T09:14:39.744 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:14:39.744 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/72533,v1:127.0.0.1:6825/72533]", 2021-05-12T09:14:39.744 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/72533,v1:127.0.0.1:6823/72533]", 2021-05-12T09:14:39.744 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:14:39.744 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:14:39.745 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:14:39.745 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:14:39.745 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:14:39.745 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:14:39.746 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:14:39.746 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:14:39.746 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:14:39.746 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T09:14:39.746 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:14:39.747 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:14:39.747 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:14:39.747 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.747 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.747 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_clean_epochs": { 2021-05-12T09:14:39.748 INFO:tasks.workunit.client.0.smithi028.stdout: "min_last_epoch_clean": 0, 2021-05-12T09:14:39.748 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": { 2021-05-12T09:14:39.748 INFO:tasks.workunit.client.0.smithi028.stdout: "per_pool": [] 2021-05-12T09:14:39.748 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.748 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_epochs": [ 2021-05-12T09:14:39.749 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.749 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:14:39.749 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 40 2021-05-12T09:14:39.749 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.750 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.750 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:14:39.750 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 42 2021-05-12T09:14:39.750 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.750 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.751 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:14:39.751 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 15 2021-05-12T09:14:39.751 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.751 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.751 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.752 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_first_committed": 1, 2021-05-12T09:14:39.752 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_last_committed": 46, 2021-05-12T09:14:39.752 INFO:tasks.workunit.client.0.smithi028.stdout: "crushmap": { 2021-05-12T09:14:39.752 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": [ 2021-05-12T09:14:39.752 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.753 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:14:39.753 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.0", 2021-05-12T09:14:39.753 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:14:39.753 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.754 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.754 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:14:39.754 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.1", 2021-05-12T09:14:39.754 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:14:39.754 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.755 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.755 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:14:39.755 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.2", 2021-05-12T09:14:39.755 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:14:39.755 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.756 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.756 INFO:tasks.workunit.client.0.smithi028.stdout: "types": [ 2021-05-12T09:14:39.756 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.756 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 0, 2021-05-12T09:14:39.757 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd" 2021-05-12T09:14:39.757 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.757 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.757 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:14:39.757 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "host" 2021-05-12T09:14:39.758 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.758 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.758 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 2, 2021-05-12T09:14:39.758 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "chassis" 2021-05-12T09:14:39.759 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.759 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.759 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 3, 2021-05-12T09:14:39.759 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "rack" 2021-05-12T09:14:39.759 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.760 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.760 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 4, 2021-05-12T09:14:39.760 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "row" 2021-05-12T09:14:39.760 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.760 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.761 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 5, 2021-05-12T09:14:39.761 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "pdu" 2021-05-12T09:14:39.761 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.761 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.761 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 6, 2021-05-12T09:14:39.762 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "pod" 2021-05-12T09:14:39.762 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.762 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.762 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 7, 2021-05-12T09:14:39.763 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "room" 2021-05-12T09:14:39.763 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.763 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.763 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 8, 2021-05-12T09:14:39.763 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "datacenter" 2021-05-12T09:14:39.764 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.764 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.764 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 9, 2021-05-12T09:14:39.764 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "zone" 2021-05-12T09:14:39.764 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.765 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.765 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 10, 2021-05-12T09:14:39.765 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "region" 2021-05-12T09:14:39.765 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.766 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.766 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:14:39.766 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "root" 2021-05-12T09:14:39.766 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.766 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.767 INFO:tasks.workunit.client.0.smithi028.stdout: "buckets": [ 2021-05-12T09:14:39.767 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.767 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -1, 2021-05-12T09:14:39.767 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "default", 2021-05-12T09:14:39.767 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:14:39.768 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "root", 2021-05-12T09:14:39.768 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:14:39.768 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:14:39.768 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:14:39.769 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:14:39.769 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.769 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -3, 2021-05-12T09:14:39.769 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:14:39.769 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:14:39.770 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.770 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.770 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.770 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.770 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -2, 2021-05-12T09:14:39.771 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "default~hdd", 2021-05-12T09:14:39.771 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:14:39.771 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "root", 2021-05-12T09:14:39.771 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:14:39.772 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:14:39.772 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:14:39.772 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:14:39.772 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.772 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -4, 2021-05-12T09:14:39.773 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:14:39.773 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:14:39.773 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.773 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.773 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.774 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.774 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -3, 2021-05-12T09:14:39.774 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "smithi028", 2021-05-12T09:14:39.774 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:14:39.775 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "host", 2021-05-12T09:14:39.775 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:14:39.775 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:14:39.775 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:14:39.775 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:14:39.776 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.776 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:14:39.776 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:14:39.776 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:14:39.776 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.777 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.777 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:14:39.777 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:14:39.777 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 1 2021-05-12T09:14:39.778 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.778 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.778 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:14:39.778 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:14:39.778 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 2 2021-05-12T09:14:39.779 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.779 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.779 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.779 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.779 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -4, 2021-05-12T09:14:39.780 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "smithi028~hdd", 2021-05-12T09:14:39.780 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:14:39.780 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "host", 2021-05-12T09:14:39.780 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:14:39.781 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:14:39.781 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:14:39.781 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:14:39.781 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.781 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:14:39.782 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:14:39.782 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:14:39.782 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.782 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.782 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:14:39.783 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:14:39.783 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 1 2021-05-12T09:14:39.783 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.783 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.783 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:14:39.784 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:14:39.784 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 2 2021-05-12T09:14:39.784 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.784 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.785 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.785 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.785 INFO:tasks.workunit.client.0.smithi028.stdout: "rules": [ 2021-05-12T09:14:39.785 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.786 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_id": 0, 2021-05-12T09:14:39.786 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_name": "replicated_rule", 2021-05-12T09:14:39.787 INFO:tasks.workunit.client.0.smithi028.stdout: "ruleset": 0, 2021-05-12T09:14:39.787 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:14:39.787 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 1, 2021-05-12T09:14:39.787 INFO:tasks.workunit.client.0.smithi028.stdout: "max_size": 10, 2021-05-12T09:14:39.787 INFO:tasks.workunit.client.0.smithi028.stdout: "steps": [ 2021-05-12T09:14:39.788 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.788 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "take", 2021-05-12T09:14:39.788 INFO:tasks.workunit.client.0.smithi028.stdout: "item": -1, 2021-05-12T09:14:39.788 INFO:tasks.workunit.client.0.smithi028.stdout: "item_name": "default" 2021-05-12T09:14:39.789 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.789 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.789 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "choose_firstn", 2021-05-12T09:14:39.789 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 0, 2021-05-12T09:14:39.789 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "osd" 2021-05-12T09:14:39.790 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.790 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.790 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "emit" 2021-05-12T09:14:39.790 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.791 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.791 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.791 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.791 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_id": 1, 2021-05-12T09:14:39.791 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_name": "test", 2021-05-12T09:14:39.792 INFO:tasks.workunit.client.0.smithi028.stdout: "ruleset": 1, 2021-05-12T09:14:39.792 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 3, 2021-05-12T09:14:39.792 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 3, 2021-05-12T09:14:39.792 INFO:tasks.workunit.client.0.smithi028.stdout: "max_size": 3, 2021-05-12T09:14:39.793 INFO:tasks.workunit.client.0.smithi028.stdout: "steps": [ 2021-05-12T09:14:39.793 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.793 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "set_chooseleaf_tries", 2021-05-12T09:14:39.793 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 5 2021-05-12T09:14:39.793 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.794 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.794 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "set_choose_tries", 2021-05-12T09:14:39.794 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 100 2021-05-12T09:14:39.794 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.794 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.795 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "take", 2021-05-12T09:14:39.795 INFO:tasks.workunit.client.0.smithi028.stdout: "item": -1, 2021-05-12T09:14:39.795 INFO:tasks.workunit.client.0.smithi028.stdout: "item_name": "default" 2021-05-12T09:14:39.795 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.796 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.796 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "choose_indep", 2021-05-12T09:14:39.796 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 0, 2021-05-12T09:14:39.796 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "osd" 2021-05-12T09:14:39.796 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.797 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.797 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "emit" 2021-05-12T09:14:39.797 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.797 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:14:39.798 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.798 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.798 INFO:tasks.workunit.client.0.smithi028.stdout: "tunables": { 2021-05-12T09:14:39.798 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_local_tries": 0, 2021-05-12T09:14:39.798 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_local_fallback_tries": 0, 2021-05-12T09:14:39.799 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_total_tries": 50, 2021-05-12T09:14:39.799 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_descend_once": 1, 2021-05-12T09:14:39.799 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_vary_r": 1, 2021-05-12T09:14:39.799 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_stable": 1, 2021-05-12T09:14:39.799 INFO:tasks.workunit.client.0.smithi028.stdout: "straw_calc_version": 1, 2021-05-12T09:14:39.800 INFO:tasks.workunit.client.0.smithi028.stdout: "allowed_bucket_algs": 54, 2021-05-12T09:14:39.800 INFO:tasks.workunit.client.0.smithi028.stdout: "profile": "jewel", 2021-05-12T09:14:39.800 INFO:tasks.workunit.client.0.smithi028.stdout: "optimal_tunables": 1, 2021-05-12T09:14:39.800 INFO:tasks.workunit.client.0.smithi028.stdout: "legacy_tunables": 0, 2021-05-12T09:14:39.801 INFO:tasks.workunit.client.0.smithi028.stdout: "minimum_required_version": "jewel", 2021-05-12T09:14:39.801 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables": 1, 2021-05-12T09:14:39.801 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables2": 1, 2021-05-12T09:14:39.801 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v2_rules": 1, 2021-05-12T09:14:39.801 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables3": 1, 2021-05-12T09:14:39.802 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v3_rules": 0, 2021-05-12T09:14:39.802 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v4_buckets": 1, 2021-05-12T09:14:39.802 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables5": 1, 2021-05-12T09:14:39.802 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v5_rules": 0 2021-05-12T09:14:39.803 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.803 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_args": {} 2021-05-12T09:14:39.803 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.803 INFO:tasks.workunit.client.0.smithi028.stdout: "fsmap": { 2021-05-12T09:14:39.803 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 1, 2021-05-12T09:14:39.804 INFO:tasks.workunit.client.0.smithi028.stdout: "default_fscid": -1, 2021-05-12T09:14:39.804 INFO:tasks.workunit.client.0.smithi028.stdout: "compat": { 2021-05-12T09:14:39.804 INFO:tasks.workunit.client.0.smithi028.stdout: "compat": {}, 2021-05-12T09:14:39.804 INFO:tasks.workunit.client.0.smithi028.stdout: "ro_compat": {}, 2021-05-12T09:14:39.804 INFO:tasks.workunit.client.0.smithi028.stdout: "incompat": { 2021-05-12T09:14:39.805 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_1": "base v0.20", 2021-05-12T09:14:39.805 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_2": "client writeable ranges", 2021-05-12T09:14:39.805 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T09:14:39.805 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_4": "dir inode in separate object", 2021-05-12T09:14:39.806 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T09:14:39.806 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T09:14:39.806 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_8": "no anchor table", 2021-05-12T09:14:39.806 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_9": "file layout v2", 2021-05-12T09:14:39.806 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_10": "snaprealm v2" 2021-05-12T09:14:39.807 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.807 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.807 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_flags": { 2021-05-12T09:14:39.807 INFO:tasks.workunit.client.0.smithi028.stdout: "enable_multiple": true, 2021-05-12T09:14:39.807 INFO:tasks.workunit.client.0.smithi028.stdout: "ever_enabled_multiple": true 2021-05-12T09:14:39.808 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.808 INFO:tasks.workunit.client.0.smithi028.stdout: "standbys": [], 2021-05-12T09:14:39.808 INFO:tasks.workunit.client.0.smithi028.stdout: "filesystems": [] 2021-05-12T09:14:39.808 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.808 INFO:tasks.workunit.client.0.smithi028.stdout: "mdsmap_first_committed": 1, 2021-05-12T09:14:39.809 INFO:tasks.workunit.client.0.smithi028.stdout: "mdsmap_last_committed": 1, 2021-05-12T09:14:39.809 INFO:tasks.workunit.client.0.smithi028.stdout: "auth": { 2021-05-12T09:14:39.809 INFO:tasks.workunit.client.0.smithi028.stdout: "first_committed": 1, 2021-05-12T09:14:39.809 INFO:tasks.workunit.client.0.smithi028.stdout: "last_committed": 5, 2021-05-12T09:14:39.810 INFO:tasks.workunit.client.0.smithi028.stdout: "num_secrets": 3 2021-05-12T09:14:39.810 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.810 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 2, 2021-05-12T09:14:39.810 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_active": 2, 2021-05-12T09:14:39.810 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_unknown": 0, 2021-05-12T09:14:39.811 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osd": 3, 2021-05-12T09:14:39.811 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_sum": { 2021-05-12T09:14:39.811 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:14:39.811 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 507710, 2021-05-12T09:14:39.811 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 103, 2021-05-12T09:14:39.812 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 1, 2021-05-12T09:14:39.812 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 309, 2021-05-12T09:14:39.812 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:14:39.812 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:14:39.813 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 103, 2021-05-12T09:14:39.813 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:14:39.813 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:14:39.813 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 103, 2021-05-12T09:14:39.813 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:14:39.814 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 76, 2021-05-12T09:14:39.814 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 64, 2021-05-12T09:14:39.814 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 196, 2021-05-12T09:14:39.814 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 1211, 2021-05-12T09:14:39.814 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:14:39.815 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:14:39.815 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:14:39.815 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:14:39.815 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:14:39.816 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:14:39.816 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:14:39.816 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:14:39.816 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:14:39.816 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:14:39.817 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:14:39.817 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:14:39.817 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:14:39.817 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:14:39.817 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:14:39.818 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:14:39.818 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:14:39.818 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:14:39.818 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:14:39.818 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:14:39.819 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:14:39.819 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:14:39.819 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:14:39.819 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:14:39.820 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:14:39.820 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.820 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:14:39.820 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:14:39.820 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:14:39.821 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:14:39.821 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 0, 2021-05-12T09:14:39.821 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 0, 2021-05-12T09:14:39.821 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:14:39.821 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:14:39.822 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:14:39.822 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:14:39.822 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:14:39.822 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.823 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 32, 2021-05-12T09:14:39.823 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 32, 2021-05-12T09:14:39.823 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 5, 2021-05-12T09:14:39.823 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 5, 2021-05-12T09:14:39.823 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 0 2021-05-12T09:14:39.824 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.824 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_sum": { 2021-05-12T09:14:39.824 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 0, 2021-05-12T09:14:39.824 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 0, 2021-05-12T09:14:39.825 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pgs": 7, 2021-05-12T09:14:39.825 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osds": 3, 2021-05-12T09:14:39.825 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_osds": 3, 2021-05-12T09:14:39.825 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T09:14:39.825 INFO:tasks.workunit.client.0.smithi028.stdout: "kb": 314572800, 2021-05-12T09:14:39.826 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used": 21480, 2021-05-12T09:14:39.826 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_data": 3592, 2021-05-12T09:14:39.826 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_omap": 2, 2021-05-12T09:14:39.826 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_meta": 17853, 2021-05-12T09:14:39.826 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_avail": 314551320, 2021-05-12T09:14:39.827 INFO:tasks.workunit.client.0.smithi028.stdout: "statfs": { 2021-05-12T09:14:39.827 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 322122547200, 2021-05-12T09:14:39.827 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 322100551680, 2021-05-12T09:14:39.827 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:14:39.827 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 3678208, 2021-05-12T09:14:39.828 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 2977099, 2021-05-12T09:14:39.828 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:14:39.828 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:14:39.828 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:14:39.829 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 2160, 2021-05-12T09:14:39.829 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 18282384 2021-05-12T09:14:39.829 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.829 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_peers": [], 2021-05-12T09:14:39.829 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_trim_queue_len": 0, 2021-05-12T09:14:39.830 INFO:tasks.workunit.client.0.smithi028.stdout: "num_snap_trimming": 0, 2021-05-12T09:14:39.830 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shards_repaired": 0, 2021-05-12T09:14:39.830 INFO:tasks.workunit.client.0.smithi028.stdout: "op_queue_age_hist": { 2021-05-12T09:14:39.830 INFO:tasks.workunit.client.0.smithi028.stdout: "histogram": [], 2021-05-12T09:14:39.830 INFO:tasks.workunit.client.0.smithi028.stdout: "upper_bound": 1 2021-05-12T09:14:39.831 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.831 INFO:tasks.workunit.client.0.smithi028.stdout: "perf_stat": { 2021-05-12T09:14:39.831 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ms": 0, 2021-05-12T09:14:39.831 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ms": 0, 2021-05-12T09:14:39.832 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ns": 0, 2021-05-12T09:14:39.832 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ns": 0 2021-05-12T09:14:39.832 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.832 INFO:tasks.workunit.client.0.smithi028.stdout: "alerts": [], 2021-05-12T09:14:39.832 INFO:tasks.workunit.client.0.smithi028.stdout: "network_ping_times": [] 2021-05-12T09:14:39.833 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.833 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_sum_by_class": { 2021-05-12T09:14:39.833 INFO:tasks.workunit.client.0.smithi028.stdout: "hdd": { 2021-05-12T09:14:39.833 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 0, 2021-05-12T09:14:39.833 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 0, 2021-05-12T09:14:39.834 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pgs": 7, 2021-05-12T09:14:39.834 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osds": 3, 2021-05-12T09:14:39.834 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_osds": 3, 2021-05-12T09:14:39.834 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T09:14:39.835 INFO:tasks.workunit.client.0.smithi028.stdout: "kb": 314572800, 2021-05-12T09:14:39.835 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used": 21480, 2021-05-12T09:14:39.835 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_data": 3592, 2021-05-12T09:14:39.835 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_omap": 2, 2021-05-12T09:14:39.835 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_meta": 17853, 2021-05-12T09:14:39.836 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_avail": 314551320, 2021-05-12T09:14:39.836 INFO:tasks.workunit.client.0.smithi028.stdout: "statfs": { 2021-05-12T09:14:39.836 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 322122547200, 2021-05-12T09:14:39.836 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 322100551680, 2021-05-12T09:14:39.836 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:14:39.837 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 3678208, 2021-05-12T09:14:39.837 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 2977099, 2021-05-12T09:14:39.837 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:14:39.837 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:14:39.837 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:14:39.838 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 2160, 2021-05-12T09:14:39.838 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 18282384 2021-05-12T09:14:39.838 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.838 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_peers": [], 2021-05-12T09:14:39.839 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_trim_queue_len": 0, 2021-05-12T09:14:39.839 INFO:tasks.workunit.client.0.smithi028.stdout: "num_snap_trimming": 0, 2021-05-12T09:14:39.839 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shards_repaired": 0, 2021-05-12T09:14:39.839 INFO:tasks.workunit.client.0.smithi028.stdout: "op_queue_age_hist": { 2021-05-12T09:14:39.839 INFO:tasks.workunit.client.0.smithi028.stdout: "histogram": [], 2021-05-12T09:14:39.840 INFO:tasks.workunit.client.0.smithi028.stdout: "upper_bound": 1 2021-05-12T09:14:39.840 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.840 INFO:tasks.workunit.client.0.smithi028.stdout: "perf_stat": { 2021-05-12T09:14:39.840 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ms": 0, 2021-05-12T09:14:39.840 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ms": 0, 2021-05-12T09:14:39.841 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ns": 0, 2021-05-12T09:14:39.841 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ns": 0 2021-05-12T09:14:39.841 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.841 INFO:tasks.workunit.client.0.smithi028.stdout: "alerts": [], 2021-05-12T09:14:39.842 INFO:tasks.workunit.client.0.smithi028.stdout: "network_ping_times": [] 2021-05-12T09:14:39.842 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.842 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.842 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_stats": [ 2021-05-12T09:14:39.842 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.843 INFO:tasks.workunit.client.0.smithi028.stdout: "poolid": 2, 2021-05-12T09:14:39.843 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 1, 2021-05-12T09:14:39.843 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:14:39.843 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 48430, 2021-05-12T09:14:39.844 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 101, 2021-05-12T09:14:39.844 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 1, 2021-05-12T09:14:39.844 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 303, 2021-05-12T09:14:39.844 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:14:39.844 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:14:39.845 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 101, 2021-05-12T09:14:39.845 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:14:39.845 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:14:39.845 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 101, 2021-05-12T09:14:39.845 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:14:39.846 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 0, 2021-05-12T09:14:39.846 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 0, 2021-05-12T09:14:39.846 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 101, 2021-05-12T09:14:39.846 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 101, 2021-05-12T09:14:39.847 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:14:39.847 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:14:39.847 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:14:39.847 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:14:39.847 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:14:39.848 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:14:39.848 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:14:39.848 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:14:39.848 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:14:39.849 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:14:39.849 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:14:39.849 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:14:39.849 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:14:39.849 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:14:39.850 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:14:39.850 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:14:39.850 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:14:39.850 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:14:39.850 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:14:39.851 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:14:39.851 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:14:39.851 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:14:39.851 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:14:39.852 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:14:39.852 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:14:39.852 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.852 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:14:39.852 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:14:39.853 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:14:39.853 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:14:39.853 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 1241088, 2021-05-12T09:14:39.853 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1241088, 2021-05-12T09:14:39.853 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:14:39.854 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:14:39.854 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:14:39.854 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:14:39.854 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:14:39.855 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.855 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 18, 2021-05-12T09:14:39.855 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 18, 2021-05-12T09:14:39.855 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 3, 2021-05-12T09:14:39.856 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 3, 2021-05-12T09:14:39.856 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 3 2021-05-12T09:14:39.856 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.856 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.856 INFO:tasks.workunit.client.0.smithi028.stdout: "poolid": 1, 2021-05-12T09:14:39.857 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 1, 2021-05-12T09:14:39.857 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:14:39.857 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 459280, 2021-05-12T09:14:39.857 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 2, 2021-05-12T09:14:39.857 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:14:39.858 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 6, 2021-05-12T09:14:39.858 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:14:39.858 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:14:39.858 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 2, 2021-05-12T09:14:39.859 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:14:39.859 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:14:39.859 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 2, 2021-05-12T09:14:39.859 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:14:39.859 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 76, 2021-05-12T09:14:39.860 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 64, 2021-05-12T09:14:39.860 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 95, 2021-05-12T09:14:39.860 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 1110, 2021-05-12T09:14:39.860 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:14:39.861 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:14:39.861 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:14:39.861 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:14:39.861 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:14:39.861 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:14:39.862 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:14:39.862 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:14:39.862 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:14:39.862 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:14:39.862 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:14:39.863 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:14:39.863 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:14:39.863 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:14:39.863 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:14:39.864 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:14:39.864 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:14:39.864 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:14:39.864 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:14:39.864 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:14:39.865 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:14:39.865 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:14:39.865 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:14:39.865 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:14:39.865 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:14:39.866 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.866 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:14:39.866 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:14:39.866 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:14:39.867 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:14:39.867 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 1388544, 2021-05-12T09:14:39.867 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1377840, 2021-05-12T09:14:39.867 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:14:39.868 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:14:39.868 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:14:39.868 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:14:39.868 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:14:39.868 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.869 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 14, 2021-05-12T09:14:39.869 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 14, 2021-05-12T09:14:39.869 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 2, 2021-05-12T09:14:39.869 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 2, 2021-05-12T09:14:39.869 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 3 2021-05-12T09:14:39.870 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.870 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.870 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_stats": [ 2021-05-12T09:14:39.870 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.871 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:14:39.871 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 171798691852 2021-05-12T09:14:39.871 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.871 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.871 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:14:39.872 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 180388626443 2021-05-12T09:14:39.872 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.872 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.872 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:14:39.873 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 64424509460 2021-05-12T09:14:39.873 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.873 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.873 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_by_state": [ 2021-05-12T09:14:39.873 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.874 INFO:tasks.workunit.client.0.smithi028.stdout: "state": "active+undersized+degraded", 2021-05-12T09:14:39.874 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 2 2021-05-12T09:14:39.874 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.874 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.874 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_by_osd": [ 2021-05-12T09:14:39.875 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.875 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:14:39.875 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:14:39.875 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 0, 2021-05-12T09:14:39.876 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:14:39.876 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.876 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.876 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2147483647, 2021-05-12T09:14:39.876 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:14:39.877 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 1, 2021-05-12T09:14:39.877 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:14:39.877 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.877 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.877 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:14:39.878 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 2, 2021-05-12T09:14:39.878 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 2, 2021-05-12T09:14:39.878 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:14:39.878 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.879 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.879 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:14:39.879 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:14:39.879 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 2, 2021-05-12T09:14:39.879 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:14:39.880 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.880 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.880 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": [ 2021-05-12T09:14:39.880 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.881 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 1, 2021-05-12T09:14:39.881 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": {} 2021-05-12T09:14:39.881 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.881 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:14:39.881 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 2, 2021-05-12T09:14:39.882 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": {} 2021-05-12T09:14:39.882 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.882 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:14:39.882 INFO:tasks.workunit.client.0.smithi028.stdout: "servicemap": { 2021-05-12T09:14:39.882 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 5, 2021-05-12T09:14:39.883 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:13:50.982632+0000", 2021-05-12T09:14:39.883 INFO:tasks.workunit.client.0.smithi028.stdout: "services": {} 2021-05-12T09:14:39.883 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:14:39.883 INFO:tasks.workunit.client.0.smithi028.stdout: "paxos": { 2021-05-12T09:14:39.884 INFO:tasks.workunit.client.0.smithi028.stdout: "first_committed": 1, 2021-05-12T09:14:39.884 INFO:tasks.workunit.client.0.smithi028.stdout: "last_committed": 413, 2021-05-12T09:14:39.884 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pn": 0, 2021-05-12T09:14:39.884 INFO:tasks.workunit.client.0.smithi028.stdout: "accepted_pn": 0 2021-05-12T09:14:39.885 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:14:39.885 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:14:39.887 INFO:tasks.workunit.client.0.smithi028.stderr:report 1463070786 2021-05-12T09:14:39.887 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T09:14:39.887 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2021-05-12T09:14:39.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2021-05-12T09:14:39.888 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:14:39.888 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:14:39.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:14:39.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:14:39.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:14:39.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:14:39.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 2.0 2021-05-12T09:14:39.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T09:14:39.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T09:14:39.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=0 2021-05-12T09:14:39.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T09:14:39.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2021-05-12T09:14:39.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 2.0 2021-05-12T09:14:40.521 INFO:tasks.workunit.client.0.smithi028.stderr:PG '2.0' not found 2021-05-12T09:14:41.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: _objectstore_tool_nodown: return 1 2021-05-12T09:14:41.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2021-05-12T09:14:41.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:14:41.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:14:41.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T09:14:41.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:14:41.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/0 2021-05-12T09:14:41.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:14:41.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:14:41.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:14:41.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:14:41.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-05-12T09:14:41.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-05-12T09:14:41.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:14:41.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:14:41.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:14:41.045 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:14:41.045 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:14:41.045 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:14:41.045 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:14:41.046 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:14:41.046 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:14:41.046 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:14:41.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:14:41.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:14:41.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:14:41.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:14:41.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:14:41.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:14:41.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:14:41.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:14:41.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:14:41.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/0 2021-05-12T09:14:41.049 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T09:14:41.049 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:14:41.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:14:41.050 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/0/whoami 2021-05-12T09:14:41.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T09:14:41.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:14:41.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:14:41.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:14:41.073 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:14:41.070+0000 7f9e901950c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:14:41.076 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:14:41.073+0000 7f9e901950c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:14:41.078 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:14:41.075+0000 7f9e901950c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:14:41.448 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T09:14:41.449 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:14:41.449 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:14:41.449 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:14:41.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:14:41.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:14:41.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:14:41.451 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:14:41.452 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:14:41.452 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:14:41.594 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:14:41.591+0000 7f9e901950c0 -1 Falling back to public interface 2021-05-12T09:14:41.837 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 40 up_thru 43 down_at 34 last_clean_interval [5,32) [v2:127.0.0.1:6802/76962,v1:127.0.0.1:6803/76962] [v2:127.0.0.1:6804/76962,v1:127.0.0.1:6805/76962] exists,up 211d7d1b-7ece-4928-932a-3aa488ac19a5 2021-05-12T09:14:41.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:14:41.839 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:14:41.839 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:14:41.839 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2021-05-12T09:14:41.839 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2021-05-12T09:14:41.840 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:14:41.840 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:14:41.840 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:14:41.840 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:14:41.841 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:14:41.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:14:41.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 2.0 2021-05-12T09:14:41.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T09:14:41.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T09:14:41.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=1 2021-05-12T09:14:41.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T09:14:41.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-05-12T09:14:41.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 2.0 2021-05-12T09:14:42.765 INFO:tasks.workunit.client.0.smithi028.stderr:PG '2.0' not found 2021-05-12T09:14:43.287 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: _objectstore_tool_nodown: return 1 2021-05-12T09:14:43.287 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 1 2021-05-12T09:14:43.287 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:14:43.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:14:43.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T09:14:43.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:14:43.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/1 2021-05-12T09:14:43.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:14:43.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:14:43.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:14:43.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:14:43.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-05-12T09:14:43.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-05-12T09:14:43.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:14:43.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:14:43.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:14:43.291 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:14:43.291 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:14:43.291 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:14:43.291 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:14:43.292 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:14:43.292 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:14:43.292 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:14:43.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:14:43.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:14:43.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:14:43.295 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:14:43.295 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:14:43.295 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:14:43.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:14:43.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:14:43.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:14:43.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/1 2021-05-12T09:14:43.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T09:14:43.297 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:14:43.298 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:14:43.299 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/1/whoami 2021-05-12T09:14:43.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T09:14:43.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:14:43.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:14:43.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:14:43.319 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:14:43.316+0000 7f783e1010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:14:43.322 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:14:43.319+0000 7f783e1010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:14:43.322 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:14:43.320+0000 7f783e1010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:14:43.686 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T09:14:43.687 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:14:43.687 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:14:43.687 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:14:43.687 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:14:43.688 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:14:43.688 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:14:43.688 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:14:43.689 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:14:43.689 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:14:43.845 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:14:43.843+0000 7f783e1010c0 -1 Falling back to public interface 2021-05-12T09:14:43.862 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:14:43.859+0000 7f9e901950c0 -1 osd.0 46 log_to_monitors {default=true} 2021-05-12T09:14:44.122 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 42 up_thru 45 down_at 35 last_clean_interval [10,32) [v2:127.0.0.1:6810/77519,v1:127.0.0.1:6811/77519] [v2:127.0.0.1:6812/77519,v1:127.0.0.1:6813/77519] exists,up 860f4a3c-1b2f-41ed-b1b5-8d8eaa20b04b 2021-05-12T09:14:44.123 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:14:44.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:14:44.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:14:44.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 2.0 2021-05-12T09:14:44.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-05-12T09:14:44.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2021-05-12T09:14:44.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local id=2 2021-05-12T09:14:44.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: shift 2021-05-12T09:14:44.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2021-05-12T09:14:44.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 2.0 2021-05-12T09:14:45.273 INFO:tasks.workunit.client.0.smithi028.stderr:PG '2.0' not found 2021-05-12T09:14:45.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: _objectstore_tool_nodown: return 1 2021-05-12T09:14:45.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-05-12T09:14:45.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2021-05-12T09:14:45.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:14:45.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:14:45.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:14:45.549 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:14:45.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 20 .1 2021-05-12T09:14:45.549 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:14:45.549 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:14:45.550 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:14:45.550 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:14:45.551 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:14:45.617 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:14:45.617 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:14:45.617 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:14:45.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:14:45.618 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:14:46.010 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:14:46.010 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:14:46.011 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:14:46.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:14:46.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:14:46.011 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:14:46.034 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:14:46.031+0000 7f783e1010c0 -1 osd.1 46 log_to_monitors {default=true} 2021-05-12T09:14:46.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=210453397507 2021-05-12T09:14:46.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 210453397507 2021-05-12T09:14:46.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-210453397507' 2021-05-12T09:14:46.176 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:14:46.176 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:14:51.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=223338299396 2021-05-12T09:14:51.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 223338299396 2021-05-12T09:14:51.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-210453397507 1-223338299396' 2021-05-12T09:14:51.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:14:51.364 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:14:51.473 INFO:tasks.workunit.client.0.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-05-12T09:14:51.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq= 2021-05-12T09:14:51.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z '' 2021-05-12T09:14:51.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: continue 2021-05-12T09:14:51.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:14:51.476 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-210453397507 2021-05-12T09:14:51.476 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:14:51.477 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:14:51.477 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-210453397507 2021-05-12T09:14:51.477 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:14:51.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=210453397507 2021-05-12T09:14:51.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 210453397507' 2021-05-12T09:14:51.479 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 210453397507 2021-05-12T09:14:51.479 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:14:51.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 210453397508 -lt 210453397507 2021-05-12T09:14:51.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:14:51.876 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-223338299396 2021-05-12T09:14:51.876 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:14:51.877 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:14:51.877 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-223338299396 2021-05-12T09:14:51.877 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:14:51.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=223338299396 2021-05-12T09:14:51.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 223338299396' 2021-05-12T09:14:51.879 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 223338299396 2021-05-12T09:14:51.879 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:14:52.266 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 1 -lt 223338299396 2021-05-12T09:14:52.266 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:14:53.267 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:14:53.268 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:14:53.668 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 223338299396 -lt 223338299396 2021-05-12T09:14:53.669 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:14:53.669 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:53.670 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:54.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:14:54.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:14:54.188 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:14:54.188 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:14:54.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:14:54.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:14:54.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:14:54.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:14:54.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:14:54.561 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:14:54.562 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:54.562 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:55.071 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:14:55.072 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T09:14:55.072 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T09:14:55.072 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T09:14:55.072 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:14:55.073 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:14:55.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:14:55.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:14:55.278 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:14:55.278 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:14:55.278 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:14:55.278 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:14:55.279 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:14:55.279 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:14:55.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:14:55.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:14:55.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:55.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:56.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:14:56.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:14:56.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:14:56.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:14:56.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:14:56.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:14:56.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:14:56.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:14:56.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:14:56.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:14:56.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:14:56.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 8 )) 2021-05-12T09:14:56.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:14:56.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T09:14:56.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:14:56.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:14:56.766 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:14:56.767 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:14:56.767 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:14:56.767 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:14:56.767 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:14:56.768 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:14:57.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:14:57.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:14:57.144 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:57.144 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:57.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:14:57.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:14:57.654 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:14:57.654 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:14:57.655 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:14:57.655 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:14:57.655 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:14:57.656 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:14:57.656 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:14:58.160 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:14:58.161 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:14:58.161 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 2 >= 8 )) 2021-05-12T09:14:58.161 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:14:58.161 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.4 2021-05-12T09:14:58.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:14:58.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:14:58.562 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:14:58.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:14:58.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:14:58.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:14:58.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:14:58.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:14:58.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:14:58.943 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:14:58.943 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:14:58.943 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:14:59.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:14:59.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:14:59.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:14:59.451 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:14:59.451 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:14:59.451 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:14:59.451 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:14:59.452 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:14:59.452 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:14:59.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:14:59.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:14:59.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 3 >= 8 )) 2021-05-12T09:14:59.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:14:59.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.8 2021-05-12T09:15:00.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:15:00.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:15:00.762 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:15:00.763 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:15:00.763 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:15:00.763 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:15:00.763 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:15:00.763 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:15:01.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:15:01.151 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:15:01.152 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:15:01.152 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:15:01.664 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:15:01.664 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:15:01.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:15:01.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:15:01.666 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:15:01.666 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:15:01.667 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:15:01.667 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:15:01.667 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:15:02.168 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:15:02.169 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:15:02.169 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 4 >= 8 )) 2021-05-12T09:15:02.169 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:15:02.169 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 1.6 2021-05-12T09:15:03.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:15:03.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:15:03.771 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:15:03.771 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:15:03.771 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:15:03.771 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:15:03.772 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:15:03.772 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:15:04.158 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:15:04.158 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:15:04.159 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:15:04.159 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:15:04.664 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:15:04.664 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:15:04.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:15:04.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:15:04.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:15:04.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:15:04.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:15:04.666 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:15:04.666 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:15:05.167 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:15:05.167 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:15:05.168 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 5 >= 8 )) 2021-05-12T09:15:05.168 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:15:05.168 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 3.2 2021-05-12T09:15:08.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:15:08.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:15:08.370 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:15:08.370 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:15:08.370 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:15:08.371 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:15:08.371 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:15:08.371 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:15:08.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:15:08.761 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:15:08.762 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:15:08.762 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:15:09.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:15:09.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:15:09.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:15:09.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:15:09.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:15:09.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:15:09.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:15:09.271 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:15:09.271 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:15:09.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:15:09.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:15:09.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 6 >= 8 )) 2021-05-12T09:15:09.776 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:15:09.776 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 6.4 2021-05-12T09:15:16.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:15:16.176 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:15:16.177 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:15:16.177 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:15:16.178 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:15:16.178 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:15:16.178 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:15:16.178 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:15:16.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:15:16.565 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:15:16.565 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:15:16.566 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:15:17.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:15:17.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:15:17.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:15:17.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:15:17.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:15:17.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:15:17.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:15:17.084 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:15:17.084 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:15:17.589 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:15:17.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:15:17.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 7 >= 8 )) 2021-05-12T09:15:17.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:15:17.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 7.3 2021-05-12T09:15:24.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:15:24.891 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:15:24.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:15:24.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:15:24.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:15:24.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:15:24.893 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:15:24.893 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:15:25.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:15:25.282 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:15:25.283 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:15:25.283 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:15:25.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:15:25.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:15:25.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:15:25.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:15:25.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:15:25.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:15:25.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:15:25.793 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:15:25.793 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:15:26.303 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:15:26.303 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:15:26.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 8 >= 8 )) 2021-05-12T09:15:26.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: ceph report 2021-05-12T09:15:26.799 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:15:26.799 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_fingerprint": "29208aba-34ab-4a89-ac1b-0d5e6428bb6d", 2021-05-12T09:15:26.800 INFO:tasks.workunit.client.0.smithi028.stdout: "version": "17.0.0-4095-ga8c673d5", 2021-05-12T09:15:26.800 INFO:tasks.workunit.client.0.smithi028.stdout: "commit": "a8c673d555488beec9ccda5812ba822e69c12ea9", 2021-05-12T09:15:26.800 INFO:tasks.workunit.client.0.smithi028.stdout: "timestamp": "2021-05-12T09:15:26.796720+0000", 2021-05-12T09:15:26.800 INFO:tasks.workunit.client.0.smithi028.stdout: "tag": "", 2021-05-12T09:15:26.801 INFO:tasks.workunit.client.0.smithi028.stdout: "health": { 2021-05-12T09:15:26.801 INFO:tasks.workunit.client.0.smithi028.stdout: "status": "HEALTH_WARN", 2021-05-12T09:15:26.801 INFO:tasks.workunit.client.0.smithi028.stdout: "checks": { 2021-05-12T09:15:26.801 INFO:tasks.workunit.client.0.smithi028.stdout: "OSDMAP_FLAGS": { 2021-05-12T09:15:26.802 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:15:26.802 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:15:26.802 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "nodown,noout,noin flag(s) set", 2021-05-12T09:15:26.803 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 17 2021-05-12T09:15:26.803 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.803 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [], 2021-05-12T09:15:26.803 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:15:26.803 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.804 INFO:tasks.workunit.client.0.smithi028.stdout: "OSD_DOWN": { 2021-05-12T09:15:26.804 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:15:26.804 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:15:26.804 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "1 osds down", 2021-05-12T09:15:26.805 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 1 2021-05-12T09:15:26.805 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.805 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:15:26.805 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.805 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "osd.2 (root=default,host=smithi028) is down" 2021-05-12T09:15:26.806 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.806 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:26.806 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:15:26.806 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.807 INFO:tasks.workunit.client.0.smithi028.stdout: "PG_DEGRADED": { 2021-05-12T09:15:26.807 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:15:26.807 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:15:26.807 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "Degraded data redundancy: 103/309 objects degraded (33.333%), 2 pgs degraded", 2021-05-12T09:15:26.807 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 2 2021-05-12T09:15:26.808 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.808 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:15:26.808 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.808 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [1,0]" 2021-05-12T09:15:26.809 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.809 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.809 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "pg 2.0 is active+undersized+degraded, acting [NONE,1,0]" 2021-05-12T09:15:26.809 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.809 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:26.810 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:15:26.810 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.810 INFO:tasks.workunit.client.0.smithi028.stdout: "POOL_APP_NOT_ENABLED": { 2021-05-12T09:15:26.810 INFO:tasks.workunit.client.0.smithi028.stdout: "severity": "HEALTH_WARN", 2021-05-12T09:15:26.811 INFO:tasks.workunit.client.0.smithi028.stdout: "summary": { 2021-05-12T09:15:26.811 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "1 pool(s) do not have an application enabled", 2021-05-12T09:15:26.811 INFO:tasks.workunit.client.0.smithi028.stdout: "count": 1 2021-05-12T09:15:26.811 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.811 INFO:tasks.workunit.client.0.smithi028.stdout: "detail": [ 2021-05-12T09:15:26.812 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.812 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "application not enabled on pool 'test'" 2021-05-12T09:15:26.812 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.812 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.812 INFO:tasks.workunit.client.0.smithi028.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-05-12T09:15:26.813 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.813 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:26.813 INFO:tasks.workunit.client.0.smithi028.stdout: "muted": false 2021-05-12T09:15:26.814 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.814 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.814 INFO:tasks.workunit.client.0.smithi028.stdout: "mutes": [] 2021-05-12T09:15:26.814 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.815 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap_first_committed": 1, 2021-05-12T09:15:26.815 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap_last_committed": 1, 2021-05-12T09:15:26.815 INFO:tasks.workunit.client.0.smithi028.stdout: "monmap": { 2021-05-12T09:15:26.815 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 1, 2021-05-12T09:15:26.815 INFO:tasks.workunit.client.0.smithi028.stdout: "fsid": "8342ec9e-531f-44b4-8c8c-079d667fe908", 2021-05-12T09:15:26.816 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:12:20.498549Z", 2021-05-12T09:15:26.816 INFO:tasks.workunit.client.0.smithi028.stdout: "created": "2021-05-12T09:12:20.498549Z", 2021-05-12T09:15:26.816 INFO:tasks.workunit.client.0.smithi028.stdout: "min_mon_release": 17, 2021-05-12T09:15:26.816 INFO:tasks.workunit.client.0.smithi028.stdout: "min_mon_release_name": "quincy", 2021-05-12T09:15:26.817 INFO:tasks.workunit.client.0.smithi028.stdout: "election_strategy": 1, 2021-05-12T09:15:26.817 INFO:tasks.workunit.client.0.smithi028.stdout: "disallowed_leaders: ": "", 2021-05-12T09:15:26.817 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode": false, 2021-05-12T09:15:26.817 INFO:tasks.workunit.client.0.smithi028.stdout: "features": { 2021-05-12T09:15:26.817 INFO:tasks.workunit.client.0.smithi028.stdout: "persistent": [ 2021-05-12T09:15:26.818 INFO:tasks.workunit.client.0.smithi028.stdout: "kraken", 2021-05-12T09:15:26.818 INFO:tasks.workunit.client.0.smithi028.stdout: "luminous", 2021-05-12T09:15:26.818 INFO:tasks.workunit.client.0.smithi028.stdout: "mimic", 2021-05-12T09:15:26.818 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap-prune", 2021-05-12T09:15:26.818 INFO:tasks.workunit.client.0.smithi028.stdout: "nautilus", 2021-05-12T09:15:26.819 INFO:tasks.workunit.client.0.smithi028.stdout: "octopus", 2021-05-12T09:15:26.819 INFO:tasks.workunit.client.0.smithi028.stdout: "pacific", 2021-05-12T09:15:26.819 INFO:tasks.workunit.client.0.smithi028.stdout: "elector-pinging", 2021-05-12T09:15:26.819 INFO:tasks.workunit.client.0.smithi028.stdout: "quincy" 2021-05-12T09:15:26.820 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:26.820 INFO:tasks.workunit.client.0.smithi028.stdout: "optional": [] 2021-05-12T09:15:26.820 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.820 INFO:tasks.workunit.client.0.smithi028.stdout: "mons": [ 2021-05-12T09:15:26.820 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.821 INFO:tasks.workunit.client.0.smithi028.stdout: "rank": 0, 2021-05-12T09:15:26.821 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "a", 2021-05-12T09:15:26.821 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:15:26.821 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:15:26.821 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.822 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:15:26.822 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:7115", 2021-05-12T09:15:26.822 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 0 2021-05-12T09:15:26.822 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.823 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.823 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.823 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:7115/0", 2021-05-12T09:15:26.823 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:7115/0", 2021-05-12T09:15:26.823 INFO:tasks.workunit.client.0.smithi028.stdout: "priority": 0, 2021-05-12T09:15:26.824 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 0, 2021-05-12T09:15:26.824 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_location": "{}" 2021-05-12T09:15:26.824 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.824 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.825 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.825 INFO:tasks.workunit.client.0.smithi028.stdout: "quorum": [ 2021-05-12T09:15:26.825 INFO:tasks.workunit.client.0.smithi028.stdout: 0 2021-05-12T09:15:26.825 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:26.825 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap": { 2021-05-12T09:15:26.826 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 53, 2021-05-12T09:15:26.826 INFO:tasks.workunit.client.0.smithi028.stdout: "fsid": "8342ec9e-531f-44b4-8c8c-079d667fe908", 2021-05-12T09:15:26.826 INFO:tasks.workunit.client.0.smithi028.stdout: "created": "2021-05-12T09:12:21.349859+0000", 2021-05-12T09:15:26.826 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:14:46.575286+0000", 2021-05-12T09:15:26.827 INFO:tasks.workunit.client.0.smithi028.stdout: "last_up_change": "2021-05-12T09:14:46.407917+0000", 2021-05-12T09:15:26.827 INFO:tasks.workunit.client.0.smithi028.stdout: "last_in_change": "2021-05-12T09:13:57.628069+0000", 2021-05-12T09:15:26.827 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-05-12T09:15:26.827 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_num": 5800384, 2021-05-12T09:15:26.827 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_set": [ 2021-05-12T09:15:26.828 INFO:tasks.workunit.client.0.smithi028.stdout: "nodown", 2021-05-12T09:15:26.828 INFO:tasks.workunit.client.0.smithi028.stdout: "noin", 2021-05-12T09:15:26.828 INFO:tasks.workunit.client.0.smithi028.stdout: "noout", 2021-05-12T09:15:26.828 INFO:tasks.workunit.client.0.smithi028.stdout: "pglog_hardlimit", 2021-05-12T09:15:26.828 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snapdirs", 2021-05-12T09:15:26.829 INFO:tasks.workunit.client.0.smithi028.stdout: "recovery_deletes", 2021-05-12T09:15:26.829 INFO:tasks.workunit.client.0.smithi028.stdout: "sortbitwise" 2021-05-12T09:15:26.829 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:26.829 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_version": 12, 2021-05-12T09:15:26.830 INFO:tasks.workunit.client.0.smithi028.stdout: "full_ratio": 0.99000000953674316, 2021-05-12T09:15:26.830 INFO:tasks.workunit.client.0.smithi028.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-05-12T09:15:26.830 INFO:tasks.workunit.client.0.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2021-05-12T09:15:26.830 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_snapshot": "", 2021-05-12T09:15:26.830 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_max": 2, 2021-05-12T09:15:26.831 INFO:tasks.workunit.client.0.smithi028.stdout: "max_osd": 3, 2021-05-12T09:15:26.831 INFO:tasks.workunit.client.0.smithi028.stdout: "require_min_compat_client": "luminous", 2021-05-12T09:15:26.831 INFO:tasks.workunit.client.0.smithi028.stdout: "min_compat_client": "jewel", 2021-05-12T09:15:26.831 INFO:tasks.workunit.client.0.smithi028.stdout: "require_osd_release": "quincy", 2021-05-12T09:15:26.831 INFO:tasks.workunit.client.0.smithi028.stdout: "pools": [ 2021-05-12T09:15:26.832 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.832 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 1, 2021-05-12T09:15:26.832 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_name": ".mgr", 2021-05-12T09:15:26.832 INFO:tasks.workunit.client.0.smithi028.stdout: "create_time": "2021-05-12T09:12:49.004998+0000", 2021-05-12T09:15:26.833 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": 1, 2021-05-12T09:15:26.833 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_names": "hashpspool", 2021-05-12T09:15:26.833 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:15:26.833 INFO:tasks.workunit.client.0.smithi028.stdout: "size": 3, 2021-05-12T09:15:26.833 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 2, 2021-05-12T09:15:26.834 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_rule": 0, 2021-05-12T09:15:26.834 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_count": 0, 2021-05-12T09:15:26.834 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_target": 0, 2021-05-12T09:15:26.834 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T09:15:26.835 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T09:15:26.835 INFO:tasks.workunit.client.0.smithi028.stdout: "object_hash": 2, 2021-05-12T09:15:26.835 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_autoscale_mode": "off", 2021-05-12T09:15:26.835 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num": 1, 2021-05-12T09:15:26.835 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num": 1, 2021-05-12T09:15:26.836 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num_target": 1, 2021-05-12T09:15:26.836 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_target": 1, 2021-05-12T09:15:26.836 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_pending": 1, 2021-05-12T09:15:26.836 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pg_merge_meta": { 2021-05-12T09:15:26.836 INFO:tasks.workunit.client.0.smithi028.stdout: "source_pgid": "0.0", 2021-05-12T09:15:26.837 INFO:tasks.workunit.client.0.smithi028.stdout: "ready_epoch": 0, 2021-05-12T09:15:26.837 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_started": 0, 2021-05-12T09:15:26.837 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": 0, 2021-05-12T09:15:26.837 INFO:tasks.workunit.client.0.smithi028.stdout: "source_version": "0'0", 2021-05-12T09:15:26.838 INFO:tasks.workunit.client.0.smithi028.stdout: "target_version": "0'0" 2021-05-12T09:15:26.838 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.838 INFO:tasks.workunit.client.0.smithi028.stdout: "last_change": "21", 2021-05-12T09:15:26.838 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend": "0", 2021-05-12T09:15:26.838 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T09:15:26.839 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T09:15:26.839 INFO:tasks.workunit.client.0.smithi028.stdout: "auid": 0, 2021-05-12T09:15:26.839 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_mode": "selfmanaged", 2021-05-12T09:15:26.839 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_seq": 0, 2021-05-12T09:15:26.839 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_epoch": 0, 2021-05-12T09:15:26.840 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_snaps": [], 2021-05-12T09:15:26.840 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps": "[]", 2021-05-12T09:15:26.840 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_bytes": 0, 2021-05-12T09:15:26.840 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_objects": 0, 2021-05-12T09:15:26.841 INFO:tasks.workunit.client.0.smithi028.stdout: "tiers": [], 2021-05-12T09:15:26.841 INFO:tasks.workunit.client.0.smithi028.stdout: "tier_of": -1, 2021-05-12T09:15:26.841 INFO:tasks.workunit.client.0.smithi028.stdout: "read_tier": -1, 2021-05-12T09:15:26.841 INFO:tasks.workunit.client.0.smithi028.stdout: "write_tier": -1, 2021-05-12T09:15:26.841 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_mode": "none", 2021-05-12T09:15:26.842 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_bytes": 0, 2021-05-12T09:15:26.842 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_objects": 0, 2021-05-12T09:15:26.842 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T09:15:26.842 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T09:15:26.843 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T09:15:26.843 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_flush_age": 0, 2021-05-12T09:15:26.843 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_evict_age": 0, 2021-05-12T09:15:26.843 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profile": "", 2021-05-12T09:15:26.843 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_params": { 2021-05-12T09:15:26.844 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "none" 2021-05-12T09:15:26.844 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.844 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_period": 0, 2021-05-12T09:15:26.844 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_count": 0, 2021-05-12T09:15:26.844 INFO:tasks.workunit.client.0.smithi028.stdout: "use_gmt_hitset": true, 2021-05-12T09:15:26.845 INFO:tasks.workunit.client.0.smithi028.stdout: "min_read_recency_for_promote": 0, 2021-05-12T09:15:26.845 INFO:tasks.workunit.client.0.smithi028.stdout: "min_write_recency_for_promote": 0, 2021-05-12T09:15:26.845 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T09:15:26.845 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_search_last_n": 0, 2021-05-12T09:15:26.846 INFO:tasks.workunit.client.0.smithi028.stdout: "grade_table": [], 2021-05-12T09:15:26.846 INFO:tasks.workunit.client.0.smithi028.stdout: "stripe_width": 0, 2021-05-12T09:15:26.846 INFO:tasks.workunit.client.0.smithi028.stdout: "expected_num_objects": 0, 2021-05-12T09:15:26.846 INFO:tasks.workunit.client.0.smithi028.stdout: "fast_read": false, 2021-05-12T09:15:26.846 INFO:tasks.workunit.client.0.smithi028.stdout: "options": { 2021-05-12T09:15:26.847 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_min": 1 2021-05-12T09:15:26.847 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.847 INFO:tasks.workunit.client.0.smithi028.stdout: "application_metadata": { 2021-05-12T09:15:26.847 INFO:tasks.workunit.client.0.smithi028.stdout: "mgr": {} 2021-05-12T09:15:26.847 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.848 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.848 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.848 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 2, 2021-05-12T09:15:26.848 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_name": "test", 2021-05-12T09:15:26.849 INFO:tasks.workunit.client.0.smithi028.stdout: "create_time": "2021-05-12T09:12:53.464789+0000", 2021-05-12T09:15:26.849 INFO:tasks.workunit.client.0.smithi028.stdout: "flags": 16389, 2021-05-12T09:15:26.849 INFO:tasks.workunit.client.0.smithi028.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2021-05-12T09:15:26.849 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 3, 2021-05-12T09:15:26.849 INFO:tasks.workunit.client.0.smithi028.stdout: "size": 3, 2021-05-12T09:15:26.850 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 2, 2021-05-12T09:15:26.850 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_rule": 1, 2021-05-12T09:15:26.850 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_count": 0, 2021-05-12T09:15:26.850 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_target": 0, 2021-05-12T09:15:26.851 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2021-05-12T09:15:26.851 INFO:tasks.workunit.client.0.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-05-12T09:15:26.851 INFO:tasks.workunit.client.0.smithi028.stdout: "object_hash": 2, 2021-05-12T09:15:26.851 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_autoscale_mode": "off", 2021-05-12T09:15:26.851 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num": 1, 2021-05-12T09:15:26.852 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num": 1, 2021-05-12T09:15:26.852 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_placement_num_target": 1, 2021-05-12T09:15:26.852 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_target": 1, 2021-05-12T09:15:26.852 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_num_pending": 1, 2021-05-12T09:15:26.853 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pg_merge_meta": { 2021-05-12T09:15:26.853 INFO:tasks.workunit.client.0.smithi028.stdout: "source_pgid": "0.0", 2021-05-12T09:15:26.853 INFO:tasks.workunit.client.0.smithi028.stdout: "ready_epoch": 0, 2021-05-12T09:15:26.853 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_started": 0, 2021-05-12T09:15:26.853 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": 0, 2021-05-12T09:15:26.854 INFO:tasks.workunit.client.0.smithi028.stdout: "source_version": "0'0", 2021-05-12T09:15:26.854 INFO:tasks.workunit.client.0.smithi028.stdout: "target_version": "0'0" 2021-05-12T09:15:26.854 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.854 INFO:tasks.workunit.client.0.smithi028.stdout: "last_change": "33", 2021-05-12T09:15:26.854 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend": "0", 2021-05-12T09:15:26.855 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2021-05-12T09:15:26.855 INFO:tasks.workunit.client.0.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2021-05-12T09:15:26.855 INFO:tasks.workunit.client.0.smithi028.stdout: "auid": 0, 2021-05-12T09:15:26.855 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_mode": "pool", 2021-05-12T09:15:26.856 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_seq": 1, 2021-05-12T09:15:26.856 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_epoch": 33, 2021-05-12T09:15:26.856 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_snaps": [ 2021-05-12T09:15:26.856 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.857 INFO:tasks.workunit.client.0.smithi028.stdout: "snapid": 1, 2021-05-12T09:15:26.857 INFO:tasks.workunit.client.0.smithi028.stdout: "stamp": "2021-05-12T09:13:35.394733+0000", 2021-05-12T09:15:26.857 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "snap1" 2021-05-12T09:15:26.857 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.858 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:26.858 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps": "[]", 2021-05-12T09:15:26.858 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_bytes": 0, 2021-05-12T09:15:26.858 INFO:tasks.workunit.client.0.smithi028.stdout: "quota_max_objects": 0, 2021-05-12T09:15:26.859 INFO:tasks.workunit.client.0.smithi028.stdout: "tiers": [], 2021-05-12T09:15:26.859 INFO:tasks.workunit.client.0.smithi028.stdout: "tier_of": -1, 2021-05-12T09:15:26.859 INFO:tasks.workunit.client.0.smithi028.stdout: "read_tier": -1, 2021-05-12T09:15:26.860 INFO:tasks.workunit.client.0.smithi028.stdout: "write_tier": -1, 2021-05-12T09:15:26.860 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_mode": "none", 2021-05-12T09:15:26.860 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_bytes": 0, 2021-05-12T09:15:26.860 INFO:tasks.workunit.client.0.smithi028.stdout: "target_max_objects": 0, 2021-05-12T09:15:26.861 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-05-12T09:15:26.861 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-05-12T09:15:26.861 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2021-05-12T09:15:26.861 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_flush_age": 0, 2021-05-12T09:15:26.862 INFO:tasks.workunit.client.0.smithi028.stdout: "cache_min_evict_age": 0, 2021-05-12T09:15:26.862 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profile": "myprofile", 2021-05-12T09:15:26.862 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_params": { 2021-05-12T09:15:26.862 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "none" 2021-05-12T09:15:26.863 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.863 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_period": 0, 2021-05-12T09:15:26.863 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_count": 0, 2021-05-12T09:15:26.863 INFO:tasks.workunit.client.0.smithi028.stdout: "use_gmt_hitset": true, 2021-05-12T09:15:26.864 INFO:tasks.workunit.client.0.smithi028.stdout: "min_read_recency_for_promote": 0, 2021-05-12T09:15:26.864 INFO:tasks.workunit.client.0.smithi028.stdout: "min_write_recency_for_promote": 0, 2021-05-12T09:15:26.864 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2021-05-12T09:15:26.864 INFO:tasks.workunit.client.0.smithi028.stdout: "hit_set_search_last_n": 0, 2021-05-12T09:15:26.865 INFO:tasks.workunit.client.0.smithi028.stdout: "grade_table": [], 2021-05-12T09:15:26.865 INFO:tasks.workunit.client.0.smithi028.stdout: "stripe_width": 8192, 2021-05-12T09:15:26.865 INFO:tasks.workunit.client.0.smithi028.stdout: "expected_num_objects": 0, 2021-05-12T09:15:26.865 INFO:tasks.workunit.client.0.smithi028.stdout: "fast_read": false, 2021-05-12T09:15:26.865 INFO:tasks.workunit.client.0.smithi028.stdout: "options": {}, 2021-05-12T09:15:26.866 INFO:tasks.workunit.client.0.smithi028.stdout: "application_metadata": {} 2021-05-12T09:15:26.866 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.866 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:26.866 INFO:tasks.workunit.client.0.smithi028.stdout: "osds": [ 2021-05-12T09:15:26.867 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.867 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:15:26.867 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "211d7d1b-7ece-4928-932a-3aa488ac19a5", 2021-05-12T09:15:26.867 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 1, 2021-05-12T09:15:26.868 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:15:26.868 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:15:26.868 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:15:26.868 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 40, 2021-05-12T09:15:26.869 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 46, 2021-05-12T09:15:26.869 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 49, 2021-05-12T09:15:26.869 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 51, 2021-05-12T09:15:26.869 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 48, 2021-05-12T09:15:26.870 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:15:26.870 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:15:26.870 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:15:26.870 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.871 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:15:26.871 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6802", 2021-05-12T09:15:26.871 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 79547 2021-05-12T09:15:26.871 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.872 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.872 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:15:26.872 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6803", 2021-05-12T09:15:26.872 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 79547 2021-05-12T09:15:26.872 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.873 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.873 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.873 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:15:26.873 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:15:26.874 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.874 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:15:26.874 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6804", 2021-05-12T09:15:26.874 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 79547 2021-05-12T09:15:26.874 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.875 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.875 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:15:26.875 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6805", 2021-05-12T09:15:26.875 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 79547 2021-05-12T09:15:26.876 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.876 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.876 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.876 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:15:26.876 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:15:26.877 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.877 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:15:26.877 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6808", 2021-05-12T09:15:26.877 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 79547 2021-05-12T09:15:26.877 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.878 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.878 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:15:26.878 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6809", 2021-05-12T09:15:26.878 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 79547 2021-05-12T09:15:26.879 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.879 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.879 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.879 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:15:26.880 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:15:26.880 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.880 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:15:26.880 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6806", 2021-05-12T09:15:26.880 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 79547 2021-05-12T09:15:26.881 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.881 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.881 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:15:26.881 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6807", 2021-05-12T09:15:26.882 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 79547 2021-05-12T09:15:26.882 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.882 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.882 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.882 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6803/79547", 2021-05-12T09:15:26.883 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6805/79547", 2021-05-12T09:15:26.883 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6809/79547", 2021-05-12T09:15:26.883 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6807/79547", 2021-05-12T09:15:26.883 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:15:26.884 INFO:tasks.workunit.client.0.smithi028.stdout: "exists", 2021-05-12T09:15:26.884 INFO:tasks.workunit.client.0.smithi028.stdout: "up" 2021-05-12T09:15:26.884 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.884 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.884 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.885 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:15:26.885 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "860f4a3c-1b2f-41ed-b1b5-8d8eaa20b04b", 2021-05-12T09:15:26.885 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 1, 2021-05-12T09:15:26.885 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:15:26.886 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:15:26.886 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:15:26.886 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 42, 2021-05-12T09:15:26.886 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 46, 2021-05-12T09:15:26.886 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 52, 2021-05-12T09:15:26.887 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 52, 2021-05-12T09:15:26.887 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 51, 2021-05-12T09:15:26.887 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:15:26.887 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:15:26.888 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:15:26.888 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.888 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:15:26.888 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6810", 2021-05-12T09:15:26.888 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 80230 2021-05-12T09:15:26.889 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.889 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.889 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:15:26.889 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6811", 2021-05-12T09:15:26.890 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 80230 2021-05-12T09:15:26.890 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.890 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.890 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.891 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:15:26.891 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:15:26.891 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.891 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:15:26.892 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6812", 2021-05-12T09:15:26.892 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 80230 2021-05-12T09:15:26.892 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.892 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.892 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:15:26.893 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6813", 2021-05-12T09:15:26.893 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 80230 2021-05-12T09:15:26.893 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.893 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.893 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.894 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:15:26.894 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:15:26.894 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.894 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:15:26.895 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6816", 2021-05-12T09:15:26.895 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 80230 2021-05-12T09:15:26.895 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.895 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.895 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:15:26.896 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6817", 2021-05-12T09:15:26.896 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 80230 2021-05-12T09:15:26.896 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.896 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.897 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.897 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:15:26.897 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:15:26.897 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.897 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:15:26.898 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6814", 2021-05-12T09:15:26.898 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 80230 2021-05-12T09:15:26.898 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.898 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.898 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:15:26.899 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6815", 2021-05-12T09:15:26.899 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 80230 2021-05-12T09:15:26.899 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.899 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.900 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.900 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6811/80230", 2021-05-12T09:15:26.900 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6813/80230", 2021-05-12T09:15:26.900 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6817/80230", 2021-05-12T09:15:26.900 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6815/80230", 2021-05-12T09:15:26.901 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:15:26.901 INFO:tasks.workunit.client.0.smithi028.stdout: "exists", 2021-05-12T09:15:26.901 INFO:tasks.workunit.client.0.smithi028.stdout: "up" 2021-05-12T09:15:26.901 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.901 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.902 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.902 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:15:26.902 INFO:tasks.workunit.client.0.smithi028.stdout: "uuid": "7951a78e-ee8e-4c4f-a30b-05cc781094fa", 2021-05-12T09:15:26.903 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 0, 2021-05-12T09:15:26.903 INFO:tasks.workunit.client.0.smithi028.stdout: "in": 1, 2021-05-12T09:15:26.903 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 1, 2021-05-12T09:15:26.903 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_affinity": 1, 2021-05-12T09:15:26.904 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_begin": 0, 2021-05-12T09:15:26.904 INFO:tasks.workunit.client.0.smithi028.stdout: "last_clean_end": 0, 2021-05-12T09:15:26.904 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 15, 2021-05-12T09:15:26.904 INFO:tasks.workunit.client.0.smithi028.stdout: "up_thru": 28, 2021-05-12T09:15:26.904 INFO:tasks.workunit.client.0.smithi028.stdout: "down_at": 36, 2021-05-12T09:15:26.905 INFO:tasks.workunit.client.0.smithi028.stdout: "lost_at": 0, 2021-05-12T09:15:26.905 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addrs": { 2021-05-12T09:15:26.905 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:15:26.905 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.905 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:15:26.906 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6818", 2021-05-12T09:15:26.906 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:15:26.906 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.906 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.907 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:15:26.907 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6819", 2021-05-12T09:15:26.907 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:15:26.907 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.907 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.908 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.908 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addrs": { 2021-05-12T09:15:26.908 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:15:26.908 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.908 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:15:26.909 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6820", 2021-05-12T09:15:26.909 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:15:26.909 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.909 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.909 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:15:26.910 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6821", 2021-05-12T09:15:26.910 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:15:26.910 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.910 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.911 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.911 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addrs": { 2021-05-12T09:15:26.911 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:15:26.911 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.911 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:15:26.912 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6824", 2021-05-12T09:15:26.912 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:15:26.912 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.912 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.913 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:15:26.913 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6825", 2021-05-12T09:15:26.913 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:15:26.913 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.913 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.914 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.914 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addrs": { 2021-05-12T09:15:26.914 INFO:tasks.workunit.client.0.smithi028.stdout: "addrvec": [ 2021-05-12T09:15:26.914 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.914 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v2", 2021-05-12T09:15:26.915 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6822", 2021-05-12T09:15:26.915 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:15:26.915 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.915 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.915 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "v1", 2021-05-12T09:15:26.916 INFO:tasks.workunit.client.0.smithi028.stdout: "addr": "127.0.0.1:6823", 2021-05-12T09:15:26.916 INFO:tasks.workunit.client.0.smithi028.stdout: "nonce": 72533 2021-05-12T09:15:26.916 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.916 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.916 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.917 INFO:tasks.workunit.client.0.smithi028.stdout: "public_addr": "127.0.0.1:6819/72533", 2021-05-12T09:15:26.917 INFO:tasks.workunit.client.0.smithi028.stdout: "cluster_addr": "127.0.0.1:6821/72533", 2021-05-12T09:15:26.917 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_back_addr": "127.0.0.1:6825/72533", 2021-05-12T09:15:26.917 INFO:tasks.workunit.client.0.smithi028.stdout: "heartbeat_front_addr": "127.0.0.1:6823/72533", 2021-05-12T09:15:26.918 INFO:tasks.workunit.client.0.smithi028.stdout: "state": [ 2021-05-12T09:15:26.918 INFO:tasks.workunit.client.0.smithi028.stdout: "exists" 2021-05-12T09:15:26.918 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.918 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.918 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:26.919 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_xinfo": [ 2021-05-12T09:15:26.919 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.919 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:15:26.919 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:14:44.133770+0000", 2021-05-12T09:15:26.919 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:15:26.920 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:15:26.920 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:15:26.920 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:15:26.920 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:12:30.963646+0000", 2021-05-12T09:15:26.921 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:15:26.921 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.921 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.921 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:15:26.921 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:14:46.249019+0000", 2021-05-12T09:15:26.922 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:15:26.922 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:15:26.922 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:15:26.922 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:15:26.922 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:12:40.136304+0000", 2021-05-12T09:15:26.923 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:15:26.923 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.923 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.923 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:15:26.923 INFO:tasks.workunit.client.0.smithi028.stdout: "down_stamp": "2021-05-12T09:13:47.536721+0000", 2021-05-12T09:15:26.924 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_probability": 0, 2021-05-12T09:15:26.924 INFO:tasks.workunit.client.0.smithi028.stdout: "laggy_interval": 0, 2021-05-12T09:15:26.924 INFO:tasks.workunit.client.0.smithi028.stdout: "features": 4540138303579357183, 2021-05-12T09:15:26.924 INFO:tasks.workunit.client.0.smithi028.stdout: "old_weight": 0, 2021-05-12T09:15:26.925 INFO:tasks.workunit.client.0.smithi028.stdout: "last_purged_snaps_scrub": "2021-05-12T09:12:49.387397+0000", 2021-05-12T09:15:26.925 INFO:tasks.workunit.client.0.smithi028.stdout: "dead_epoch": 0 2021-05-12T09:15:26.925 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.925 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:26.926 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_upmap": [], 2021-05-12T09:15:26.926 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_upmap_items": [], 2021-05-12T09:15:26.926 INFO:tasks.workunit.client.0.smithi028.stdout: "pg_temp": [], 2021-05-12T09:15:26.926 INFO:tasks.workunit.client.0.smithi028.stdout: "primary_temp": [], 2021-05-12T09:15:26.926 INFO:tasks.workunit.client.0.smithi028.stdout: "blocklist": {}, 2021-05-12T09:15:26.927 INFO:tasks.workunit.client.0.smithi028.stdout: "erasure_code_profiles": { 2021-05-12T09:15:26.927 INFO:tasks.workunit.client.0.smithi028.stdout: "default": { 2021-05-12T09:15:26.927 INFO:tasks.workunit.client.0.smithi028.stdout: "k": "2", 2021-05-12T09:15:26.927 INFO:tasks.workunit.client.0.smithi028.stdout: "m": "2", 2021-05-12T09:15:26.927 INFO:tasks.workunit.client.0.smithi028.stdout: "plugin": "jerasure", 2021-05-12T09:15:26.928 INFO:tasks.workunit.client.0.smithi028.stdout: "technique": "reed_sol_van" 2021-05-12T09:15:26.928 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.928 INFO:tasks.workunit.client.0.smithi028.stdout: "myprofile": { 2021-05-12T09:15:26.928 INFO:tasks.workunit.client.0.smithi028.stdout: "crush-device-class": "", 2021-05-12T09:15:26.928 INFO:tasks.workunit.client.0.smithi028.stdout: "crush-failure-domain": "osd", 2021-05-12T09:15:26.929 INFO:tasks.workunit.client.0.smithi028.stdout: "crush-root": "default", 2021-05-12T09:15:26.929 INFO:tasks.workunit.client.0.smithi028.stdout: "jerasure-per-chunk-alignment": "false", 2021-05-12T09:15:26.929 INFO:tasks.workunit.client.0.smithi028.stdout: "k": "2", 2021-05-12T09:15:26.929 INFO:tasks.workunit.client.0.smithi028.stdout: "m": "1", 2021-05-12T09:15:26.929 INFO:tasks.workunit.client.0.smithi028.stdout: "plugin": "jerasure", 2021-05-12T09:15:26.930 INFO:tasks.workunit.client.0.smithi028.stdout: "technique": "reed_sol_van", 2021-05-12T09:15:26.930 INFO:tasks.workunit.client.0.smithi028.stdout: "w": "8" 2021-05-12T09:15:26.930 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.930 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.931 INFO:tasks.workunit.client.0.smithi028.stdout: "removed_snaps_queue": [], 2021-05-12T09:15:26.931 INFO:tasks.workunit.client.0.smithi028.stdout: "new_removed_snaps": [], 2021-05-12T09:15:26.931 INFO:tasks.workunit.client.0.smithi028.stdout: "new_purged_snaps": [], 2021-05-12T09:15:26.931 INFO:tasks.workunit.client.0.smithi028.stdout: "crush_node_flags": {}, 2021-05-12T09:15:26.931 INFO:tasks.workunit.client.0.smithi028.stdout: "device_class_flags": {}, 2021-05-12T09:15:26.932 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode": { 2021-05-12T09:15:26.932 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode_enabled": false, 2021-05-12T09:15:26.932 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_bucket_count": 0, 2021-05-12T09:15:26.932 INFO:tasks.workunit.client.0.smithi028.stdout: "degraded_stretch_mode": 0, 2021-05-12T09:15:26.932 INFO:tasks.workunit.client.0.smithi028.stdout: "recovering_stretch_mode": 0, 2021-05-12T09:15:26.933 INFO:tasks.workunit.client.0.smithi028.stdout: "stretch_mode_bucket": 0 2021-05-12T09:15:26.933 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.933 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.934 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_metadata": [ 2021-05-12T09:15:26.934 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.934 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:15:26.934 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:15:26.934 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6804/79547,v1:127.0.0.1:6805/79547]", 2021-05-12T09:15:26.935 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:15:26.935 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:15:26.935 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:15:26.935 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:15:26.936 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:15:26.936 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:15:26.936 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:15:26.936 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:15:26.936 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:15:26.937 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-05-12T09:15:26.937 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:15:26.937 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:15:26.937 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:15:26.937 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:15:26.938 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:15:26.938 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:15:26.938 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:15:26.938 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:15:26.939 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:15:26.939 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:15:26.939 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:15:26.939 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:15:26.939 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:15:26.940 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:15:26.940 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:15:26.940 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6802/79547,v1:127.0.0.1:6803/79547]", 2021-05-12T09:15:26.940 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:15:26.940 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/79547,v1:127.0.0.1:6809/79547]", 2021-05-12T09:15:26.941 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/79547,v1:127.0.0.1:6807/79547]", 2021-05-12T09:15:26.941 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:15:26.941 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:15:26.941 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:15:26.941 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:15:26.942 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:15:26.942 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:15:26.942 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:15:26.943 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:15:26.943 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:15:26.943 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/0", 2021-05-12T09:15:26.943 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:15:26.943 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:15:26.944 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:15:26.944 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.944 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.944 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:15:26.944 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:15:26.945 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6812/80230,v1:127.0.0.1:6813/80230]", 2021-05-12T09:15:26.945 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:15:26.945 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:15:26.945 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:15:26.945 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:15:26.946 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:15:26.946 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:15:26.946 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:15:26.946 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:15:26.946 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:15:26.947 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-05-12T09:15:26.947 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:15:26.947 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:15:26.947 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:15:26.947 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:15:26.948 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:15:26.948 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:15:26.948 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:15:26.948 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:15:26.949 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:15:26.949 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:15:26.949 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:15:26.949 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:15:26.949 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:15:26.950 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:15:26.950 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:15:26.950 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6810/80230,v1:127.0.0.1:6811/80230]", 2021-05-12T09:15:26.950 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:15:26.950 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/80230,v1:127.0.0.1:6817/80230]", 2021-05-12T09:15:26.951 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/80230,v1:127.0.0.1:6815/80230]", 2021-05-12T09:15:26.951 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:15:26.951 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:15:26.951 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:15:26.951 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:15:26.952 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:15:26.952 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:15:26.952 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:15:26.952 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:15:26.953 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:15:26.953 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/1", 2021-05-12T09:15:26.953 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:15:26.953 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:15:26.953 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:15:26.954 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.954 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.954 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:15:26.954 INFO:tasks.workunit.client.0.smithi028.stdout: "arch": "x86_64", 2021-05-12T09:15:26.954 INFO:tasks.workunit.client.0.smithi028.stdout: "back_addr": "[v2:127.0.0.1:6820/72533,v1:127.0.0.1:6821/72533]", 2021-05-12T09:15:26.955 INFO:tasks.workunit.client.0.smithi028.stdout: "back_iface": "", 2021-05-12T09:15:26.955 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs": "1", 2021-05-12T09:15:26.955 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_db": "0", 2021-05-12T09:15:26.955 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_dedicated_wal": "0", 2021-05-12T09:15:26.955 INFO:tasks.workunit.client.0.smithi028.stdout: "bluefs_single_shared_device": "1", 2021-05-12T09:15:26.956 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:15:26.956 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:15:26.956 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:15:26.956 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:15:26.956 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-05-12T09:15:26.957 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:15:26.957 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:15:26.957 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:15:26.957 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:15:26.957 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_release": "quincy", 2021-05-12T09:15:26.958 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version": "ceph version 17.0.0-4095-ga8c673d5 (a8c673d555488beec9ccda5812ba822e69c12ea9) quincy (dev)", 2021-05-12T09:15:26.958 INFO:tasks.workunit.client.0.smithi028.stdout: "ceph_version_short": "17.0.0-4095-ga8c673d5", 2021-05-12T09:15:26.958 INFO:tasks.workunit.client.0.smithi028.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2021-05-12T09:15:26.958 INFO:tasks.workunit.client.0.smithi028.stdout: "default_device_class": "hdd", 2021-05-12T09:15:26.959 INFO:tasks.workunit.client.0.smithi028.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W4J30L", 2021-05-12T09:15:26.959 INFO:tasks.workunit.client.0.smithi028.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-05-12T09:15:26.959 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": "sda", 2021-05-12T09:15:26.959 INFO:tasks.workunit.client.0.smithi028.stdout: "distro": "centos", 2021-05-12T09:15:26.959 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_description": "CentOS Linux 8", 2021-05-12T09:15:26.960 INFO:tasks.workunit.client.0.smithi028.stdout: "distro_version": "8", 2021-05-12T09:15:26.960 INFO:tasks.workunit.client.0.smithi028.stdout: "front_addr": "[v2:127.0.0.1:6818/72533,v1:127.0.0.1:6819/72533]", 2021-05-12T09:15:26.960 INFO:tasks.workunit.client.0.smithi028.stdout: "front_iface": "", 2021-05-12T09:15:26.960 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/72533,v1:127.0.0.1:6825/72533]", 2021-05-12T09:15:26.960 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/72533,v1:127.0.0.1:6823/72533]", 2021-05-12T09:15:26.961 INFO:tasks.workunit.client.0.smithi028.stdout: "hostname": "smithi028", 2021-05-12T09:15:26.961 INFO:tasks.workunit.client.0.smithi028.stdout: "journal_rotational": "1", 2021-05-12T09:15:26.961 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_description": "#1 SMP Thu Apr 8 19:01:30 UTC 2021", 2021-05-12T09:15:26.961 INFO:tasks.workunit.client.0.smithi028.stdout: "kernel_version": "4.18.0-240.22.1.el8_3.x86_64", 2021-05-12T09:15:26.961 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_swap_kb": "0", 2021-05-12T09:15:26.962 INFO:tasks.workunit.client.0.smithi028.stdout: "mem_total_kb": "32618232", 2021-05-12T09:15:26.962 INFO:tasks.workunit.client.0.smithi028.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-05-12T09:15:26.962 INFO:tasks.workunit.client.0.smithi028.stdout: "objectstore_numa_unknown_devices": "sda", 2021-05-12T09:15:26.962 INFO:tasks.workunit.client.0.smithi028.stdout: "os": "Linux", 2021-05-12T09:15:26.963 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_data": "td/divergent-priors/2", 2021-05-12T09:15:26.963 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:15:26.963 INFO:tasks.workunit.client.0.smithi028.stdout: "osdspec_affinity": "", 2021-05-12T09:15:26.963 INFO:tasks.workunit.client.0.smithi028.stdout: "rotational": "1" 2021-05-12T09:15:26.963 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.964 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:26.964 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_clean_epochs": { 2021-05-12T09:15:26.964 INFO:tasks.workunit.client.0.smithi028.stdout: "min_last_epoch_clean": 0, 2021-05-12T09:15:26.964 INFO:tasks.workunit.client.0.smithi028.stdout: "last_epoch_clean": { 2021-05-12T09:15:26.964 INFO:tasks.workunit.client.0.smithi028.stdout: "per_pool": [] 2021-05-12T09:15:26.965 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.965 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_epochs": [ 2021-05-12T09:15:26.965 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.965 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:15:26.965 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 49 2021-05-12T09:15:26.966 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.966 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.966 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:15:26.966 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 53 2021-05-12T09:15:26.966 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.967 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.967 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:15:26.967 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 15 2021-05-12T09:15:26.967 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.967 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.968 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.968 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_first_committed": 1, 2021-05-12T09:15:26.968 INFO:tasks.workunit.client.0.smithi028.stdout: "osdmap_last_committed": 53, 2021-05-12T09:15:26.968 INFO:tasks.workunit.client.0.smithi028.stdout: "crushmap": { 2021-05-12T09:15:26.968 INFO:tasks.workunit.client.0.smithi028.stdout: "devices": [ 2021-05-12T09:15:26.969 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.969 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:15:26.969 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.0", 2021-05-12T09:15:26.969 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:15:26.969 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.970 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.970 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:15:26.970 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.1", 2021-05-12T09:15:26.970 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:15:26.970 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.971 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.971 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:15:26.971 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd.2", 2021-05-12T09:15:26.971 INFO:tasks.workunit.client.0.smithi028.stdout: "class": "hdd" 2021-05-12T09:15:26.972 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.972 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:26.972 INFO:tasks.workunit.client.0.smithi028.stdout: "types": [ 2021-05-12T09:15:26.972 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.972 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 0, 2021-05-12T09:15:26.973 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "osd" 2021-05-12T09:15:26.973 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.973 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.973 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:15:26.973 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "host" 2021-05-12T09:15:26.974 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.974 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.974 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 2, 2021-05-12T09:15:26.974 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "chassis" 2021-05-12T09:15:26.974 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.975 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.975 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 3, 2021-05-12T09:15:26.975 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "rack" 2021-05-12T09:15:26.975 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.975 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.976 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 4, 2021-05-12T09:15:26.976 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "row" 2021-05-12T09:15:26.976 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.976 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.976 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 5, 2021-05-12T09:15:26.977 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "pdu" 2021-05-12T09:15:26.977 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.977 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.977 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 6, 2021-05-12T09:15:26.977 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "pod" 2021-05-12T09:15:26.978 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.978 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.978 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 7, 2021-05-12T09:15:26.978 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "room" 2021-05-12T09:15:26.978 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.979 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.979 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 8, 2021-05-12T09:15:26.979 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "datacenter" 2021-05-12T09:15:26.979 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.979 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.980 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 9, 2021-05-12T09:15:26.980 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "zone" 2021-05-12T09:15:26.980 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.980 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.980 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 10, 2021-05-12T09:15:26.981 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "region" 2021-05-12T09:15:26.981 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.981 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.981 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:15:26.981 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "root" 2021-05-12T09:15:26.982 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.982 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:26.982 INFO:tasks.workunit.client.0.smithi028.stdout: "buckets": [ 2021-05-12T09:15:26.982 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.982 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -1, 2021-05-12T09:15:26.983 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "default", 2021-05-12T09:15:26.983 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:15:26.983 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "root", 2021-05-12T09:15:26.983 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:15:26.984 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:15:26.984 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:15:26.984 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:15:26.984 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.984 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -3, 2021-05-12T09:15:26.985 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:15:26.985 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:15:26.985 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.985 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.985 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.986 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.986 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -2, 2021-05-12T09:15:26.986 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "default~hdd", 2021-05-12T09:15:26.986 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 11, 2021-05-12T09:15:26.986 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "root", 2021-05-12T09:15:26.987 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:15:26.987 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:15:26.987 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:15:26.987 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:15:26.987 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.988 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -4, 2021-05-12T09:15:26.988 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:15:26.988 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:15:26.988 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.988 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.989 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.989 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.989 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -3, 2021-05-12T09:15:26.989 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "smithi028", 2021-05-12T09:15:26.989 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:15:26.990 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "host", 2021-05-12T09:15:26.990 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:15:26.990 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:15:26.990 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:15:26.990 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:15:26.991 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.991 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:15:26.991 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:15:26.991 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:15:26.991 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.992 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.992 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:15:26.992 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:15:26.992 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 1 2021-05-12T09:15:26.992 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.993 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.993 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:15:26.993 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:15:26.993 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 2 2021-05-12T09:15:26.994 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.994 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.994 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.994 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.994 INFO:tasks.workunit.client.0.smithi028.stdout: "id": -4, 2021-05-12T09:15:26.995 INFO:tasks.workunit.client.0.smithi028.stdout: "name": "smithi028~hdd", 2021-05-12T09:15:26.995 INFO:tasks.workunit.client.0.smithi028.stdout: "type_id": 1, 2021-05-12T09:15:26.995 INFO:tasks.workunit.client.0.smithi028.stdout: "type_name": "host", 2021-05-12T09:15:26.995 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 19206, 2021-05-12T09:15:26.995 INFO:tasks.workunit.client.0.smithi028.stdout: "alg": "straw2", 2021-05-12T09:15:26.996 INFO:tasks.workunit.client.0.smithi028.stdout: "hash": "rjenkins1", 2021-05-12T09:15:26.996 INFO:tasks.workunit.client.0.smithi028.stdout: "items": [ 2021-05-12T09:15:26.996 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.996 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 0, 2021-05-12T09:15:26.996 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:15:26.997 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 0 2021-05-12T09:15:26.997 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.997 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.997 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 1, 2021-05-12T09:15:26.997 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:15:26.998 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 1 2021-05-12T09:15:26.998 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:26.998 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:26.998 INFO:tasks.workunit.client.0.smithi028.stdout: "id": 2, 2021-05-12T09:15:26.998 INFO:tasks.workunit.client.0.smithi028.stdout: "weight": 6402, 2021-05-12T09:15:26.999 INFO:tasks.workunit.client.0.smithi028.stdout: "pos": 2 2021-05-12T09:15:26.999 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.999 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:26.999 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:26.999 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:27.000 INFO:tasks.workunit.client.0.smithi028.stdout: "rules": [ 2021-05-12T09:15:27.000 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.000 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_id": 0, 2021-05-12T09:15:27.000 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_name": "replicated_rule", 2021-05-12T09:15:27.000 INFO:tasks.workunit.client.0.smithi028.stdout: "ruleset": 0, 2021-05-12T09:15:27.001 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 1, 2021-05-12T09:15:27.001 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 1, 2021-05-12T09:15:27.001 INFO:tasks.workunit.client.0.smithi028.stdout: "max_size": 10, 2021-05-12T09:15:27.001 INFO:tasks.workunit.client.0.smithi028.stdout: "steps": [ 2021-05-12T09:15:27.001 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.002 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "take", 2021-05-12T09:15:27.002 INFO:tasks.workunit.client.0.smithi028.stdout: "item": -1, 2021-05-12T09:15:27.002 INFO:tasks.workunit.client.0.smithi028.stdout: "item_name": "default" 2021-05-12T09:15:27.002 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.003 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.003 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "choose_firstn", 2021-05-12T09:15:27.003 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 0, 2021-05-12T09:15:27.003 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "osd" 2021-05-12T09:15:27.003 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.004 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.004 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "emit" 2021-05-12T09:15:27.004 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:27.004 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:27.004 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.005 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.005 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_id": 1, 2021-05-12T09:15:27.005 INFO:tasks.workunit.client.0.smithi028.stdout: "rule_name": "test", 2021-05-12T09:15:27.005 INFO:tasks.workunit.client.0.smithi028.stdout: "ruleset": 1, 2021-05-12T09:15:27.005 INFO:tasks.workunit.client.0.smithi028.stdout: "type": 3, 2021-05-12T09:15:27.006 INFO:tasks.workunit.client.0.smithi028.stdout: "min_size": 3, 2021-05-12T09:15:27.006 INFO:tasks.workunit.client.0.smithi028.stdout: "max_size": 3, 2021-05-12T09:15:27.006 INFO:tasks.workunit.client.0.smithi028.stdout: "steps": [ 2021-05-12T09:15:27.006 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.006 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "set_chooseleaf_tries", 2021-05-12T09:15:27.007 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 5 2021-05-12T09:15:27.007 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.007 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.007 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "set_choose_tries", 2021-05-12T09:15:27.007 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 100 2021-05-12T09:15:27.008 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.008 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.008 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "take", 2021-05-12T09:15:27.008 INFO:tasks.workunit.client.0.smithi028.stdout: "item": -1, 2021-05-12T09:15:27.008 INFO:tasks.workunit.client.0.smithi028.stdout: "item_name": "default" 2021-05-12T09:15:27.009 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.009 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.009 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "choose_indep", 2021-05-12T09:15:27.009 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 0, 2021-05-12T09:15:27.009 INFO:tasks.workunit.client.0.smithi028.stdout: "type": "osd" 2021-05-12T09:15:27.010 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.010 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.010 INFO:tasks.workunit.client.0.smithi028.stdout: "op": "emit" 2021-05-12T09:15:27.010 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:27.011 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:15:27.011 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:27.011 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:27.011 INFO:tasks.workunit.client.0.smithi028.stdout: "tunables": { 2021-05-12T09:15:27.011 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_local_tries": 0, 2021-05-12T09:15:27.012 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_local_fallback_tries": 0, 2021-05-12T09:15:27.012 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_total_tries": 50, 2021-05-12T09:15:27.012 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_descend_once": 1, 2021-05-12T09:15:27.012 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_vary_r": 1, 2021-05-12T09:15:27.012 INFO:tasks.workunit.client.0.smithi028.stdout: "chooseleaf_stable": 1, 2021-05-12T09:15:27.013 INFO:tasks.workunit.client.0.smithi028.stdout: "straw_calc_version": 1, 2021-05-12T09:15:27.013 INFO:tasks.workunit.client.0.smithi028.stdout: "allowed_bucket_algs": 54, 2021-05-12T09:15:27.013 INFO:tasks.workunit.client.0.smithi028.stdout: "profile": "jewel", 2021-05-12T09:15:27.013 INFO:tasks.workunit.client.0.smithi028.stdout: "optimal_tunables": 1, 2021-05-12T09:15:27.013 INFO:tasks.workunit.client.0.smithi028.stdout: "legacy_tunables": 0, 2021-05-12T09:15:27.014 INFO:tasks.workunit.client.0.smithi028.stdout: "minimum_required_version": "jewel", 2021-05-12T09:15:27.014 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables": 1, 2021-05-12T09:15:27.014 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables2": 1, 2021-05-12T09:15:27.014 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v2_rules": 1, 2021-05-12T09:15:27.014 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables3": 1, 2021-05-12T09:15:27.015 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v3_rules": 0, 2021-05-12T09:15:27.015 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v4_buckets": 1, 2021-05-12T09:15:27.015 INFO:tasks.workunit.client.0.smithi028.stdout: "require_feature_tunables5": 1, 2021-05-12T09:15:27.015 INFO:tasks.workunit.client.0.smithi028.stdout: "has_v5_rules": 0 2021-05-12T09:15:27.015 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.016 INFO:tasks.workunit.client.0.smithi028.stdout: "choose_args": {} 2021-05-12T09:15:27.016 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.016 INFO:tasks.workunit.client.0.smithi028.stdout: "fsmap": { 2021-05-12T09:15:27.016 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 1, 2021-05-12T09:15:27.016 INFO:tasks.workunit.client.0.smithi028.stdout: "default_fscid": -1, 2021-05-12T09:15:27.017 INFO:tasks.workunit.client.0.smithi028.stdout: "compat": { 2021-05-12T09:15:27.017 INFO:tasks.workunit.client.0.smithi028.stdout: "compat": {}, 2021-05-12T09:15:27.017 INFO:tasks.workunit.client.0.smithi028.stdout: "ro_compat": {}, 2021-05-12T09:15:27.017 INFO:tasks.workunit.client.0.smithi028.stdout: "incompat": { 2021-05-12T09:15:27.017 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_1": "base v0.20", 2021-05-12T09:15:27.018 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_2": "client writeable ranges", 2021-05-12T09:15:27.018 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_3": "default file layouts on dirs", 2021-05-12T09:15:27.018 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_4": "dir inode in separate object", 2021-05-12T09:15:27.018 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_5": "mds uses versioned encoding", 2021-05-12T09:15:27.018 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_6": "dirfrag is stored in omap", 2021-05-12T09:15:27.019 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_8": "no anchor table", 2021-05-12T09:15:27.019 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_9": "file layout v2", 2021-05-12T09:15:27.019 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_10": "snaprealm v2" 2021-05-12T09:15:27.019 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:27.019 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.020 INFO:tasks.workunit.client.0.smithi028.stdout: "feature_flags": { 2021-05-12T09:15:27.020 INFO:tasks.workunit.client.0.smithi028.stdout: "enable_multiple": true, 2021-05-12T09:15:27.020 INFO:tasks.workunit.client.0.smithi028.stdout: "ever_enabled_multiple": true 2021-05-12T09:15:27.020 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.021 INFO:tasks.workunit.client.0.smithi028.stdout: "standbys": [], 2021-05-12T09:15:27.021 INFO:tasks.workunit.client.0.smithi028.stdout: "filesystems": [] 2021-05-12T09:15:27.021 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.021 INFO:tasks.workunit.client.0.smithi028.stdout: "mdsmap_first_committed": 1, 2021-05-12T09:15:27.021 INFO:tasks.workunit.client.0.smithi028.stdout: "mdsmap_last_committed": 1, 2021-05-12T09:15:27.022 INFO:tasks.workunit.client.0.smithi028.stdout: "auth": { 2021-05-12T09:15:27.022 INFO:tasks.workunit.client.0.smithi028.stdout: "first_committed": 1, 2021-05-12T09:15:27.022 INFO:tasks.workunit.client.0.smithi028.stdout: "last_committed": 5, 2021-05-12T09:15:27.022 INFO:tasks.workunit.client.0.smithi028.stdout: "num_secrets": 3 2021-05-12T09:15:27.022 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.023 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 2, 2021-05-12T09:15:27.023 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_active": 2, 2021-05-12T09:15:27.023 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_unknown": 0, 2021-05-12T09:15:27.023 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osd": 3, 2021-05-12T09:15:27.023 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_sum": { 2021-05-12T09:15:27.024 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:15:27.024 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 507710, 2021-05-12T09:15:27.024 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 103, 2021-05-12T09:15:27.024 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 1, 2021-05-12T09:15:27.024 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 309, 2021-05-12T09:15:27.025 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:15:27.025 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:15:27.025 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 103, 2021-05-12T09:15:27.025 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:15:27.025 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:15:27.026 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 103, 2021-05-12T09:15:27.026 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:15:27.026 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 76, 2021-05-12T09:15:27.026 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 64, 2021-05-12T09:15:27.026 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 196, 2021-05-12T09:15:27.027 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 1211, 2021-05-12T09:15:27.027 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:15:27.027 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:15:27.027 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:15:27.027 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:15:27.028 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:15:27.028 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:15:27.028 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:15:27.028 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:15:27.028 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:15:27.029 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:15:27.029 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:15:27.029 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:15:27.029 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:15:27.029 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:15:27.030 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:15:27.030 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:15:27.030 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:15:27.030 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:15:27.031 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:15:27.031 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:15:27.031 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:15:27.031 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:15:27.031 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:15:27.032 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:15:27.032 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:15:27.032 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.032 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:15:27.032 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:15:27.033 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:15:27.033 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:15:27.033 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 0, 2021-05-12T09:15:27.033 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 0, 2021-05-12T09:15:27.034 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:15:27.034 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:15:27.034 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:15:27.034 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:15:27.034 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:15:27.035 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.035 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 32, 2021-05-12T09:15:27.035 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 32, 2021-05-12T09:15:27.035 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 5, 2021-05-12T09:15:27.035 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 5, 2021-05-12T09:15:27.036 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 0 2021-05-12T09:15:27.036 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.036 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_sum": { 2021-05-12T09:15:27.036 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 0, 2021-05-12T09:15:27.036 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 0, 2021-05-12T09:15:27.037 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pgs": 7, 2021-05-12T09:15:27.037 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osds": 3, 2021-05-12T09:15:27.037 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_osds": 3, 2021-05-12T09:15:27.037 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T09:15:27.037 INFO:tasks.workunit.client.0.smithi028.stdout: "kb": 314572800, 2021-05-12T09:15:27.038 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used": 22168, 2021-05-12T09:15:27.038 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_data": 3704, 2021-05-12T09:15:27.038 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_omap": 4, 2021-05-12T09:15:27.038 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_meta": 18427, 2021-05-12T09:15:27.039 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_avail": 314550632, 2021-05-12T09:15:27.039 INFO:tasks.workunit.client.0.smithi028.stdout: "statfs": { 2021-05-12T09:15:27.039 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 322122547200, 2021-05-12T09:15:27.039 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 322099847168, 2021-05-12T09:15:27.039 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:15:27.040 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 3792896, 2021-05-12T09:15:27.040 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 3029771, 2021-05-12T09:15:27.040 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:15:27.040 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:15:27.041 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:15:27.041 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 4866, 2021-05-12T09:15:27.041 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 18869502 2021-05-12T09:15:27.041 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.041 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_peers": [], 2021-05-12T09:15:27.042 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_trim_queue_len": 0, 2021-05-12T09:15:27.042 INFO:tasks.workunit.client.0.smithi028.stdout: "num_snap_trimming": 0, 2021-05-12T09:15:27.042 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shards_repaired": 0, 2021-05-12T09:15:27.042 INFO:tasks.workunit.client.0.smithi028.stdout: "op_queue_age_hist": { 2021-05-12T09:15:27.042 INFO:tasks.workunit.client.0.smithi028.stdout: "histogram": [], 2021-05-12T09:15:27.043 INFO:tasks.workunit.client.0.smithi028.stdout: "upper_bound": 1 2021-05-12T09:15:27.043 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.043 INFO:tasks.workunit.client.0.smithi028.stdout: "perf_stat": { 2021-05-12T09:15:27.043 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ms": 0, 2021-05-12T09:15:27.043 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ms": 0, 2021-05-12T09:15:27.044 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ns": 0, 2021-05-12T09:15:27.044 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ns": 0 2021-05-12T09:15:27.044 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.044 INFO:tasks.workunit.client.0.smithi028.stdout: "alerts": [], 2021-05-12T09:15:27.045 INFO:tasks.workunit.client.0.smithi028.stdout: "network_ping_times": [] 2021-05-12T09:15:27.045 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.045 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_sum_by_class": { 2021-05-12T09:15:27.045 INFO:tasks.workunit.client.0.smithi028.stdout: "hdd": { 2021-05-12T09:15:27.045 INFO:tasks.workunit.client.0.smithi028.stdout: "up_from": 0, 2021-05-12T09:15:27.046 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 0, 2021-05-12T09:15:27.046 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pgs": 7, 2021-05-12T09:15:27.046 INFO:tasks.workunit.client.0.smithi028.stdout: "num_osds": 3, 2021-05-12T09:15:27.046 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_osds": 3, 2021-05-12T09:15:27.046 INFO:tasks.workunit.client.0.smithi028.stdout: "num_per_pool_omap_osds": 3, 2021-05-12T09:15:27.047 INFO:tasks.workunit.client.0.smithi028.stdout: "kb": 314572800, 2021-05-12T09:15:27.047 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used": 22168, 2021-05-12T09:15:27.047 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_data": 3704, 2021-05-12T09:15:27.047 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_omap": 4, 2021-05-12T09:15:27.047 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_used_meta": 18427, 2021-05-12T09:15:27.048 INFO:tasks.workunit.client.0.smithi028.stdout: "kb_avail": 314550632, 2021-05-12T09:15:27.048 INFO:tasks.workunit.client.0.smithi028.stdout: "statfs": { 2021-05-12T09:15:27.048 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 322122547200, 2021-05-12T09:15:27.048 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 322099847168, 2021-05-12T09:15:27.048 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:15:27.049 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 3792896, 2021-05-12T09:15:27.049 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 3029771, 2021-05-12T09:15:27.049 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:15:27.049 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:15:27.049 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:15:27.050 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 4866, 2021-05-12T09:15:27.050 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 18869502 2021-05-12T09:15:27.050 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.050 INFO:tasks.workunit.client.0.smithi028.stdout: "hb_peers": [], 2021-05-12T09:15:27.050 INFO:tasks.workunit.client.0.smithi028.stdout: "snap_trim_queue_len": 0, 2021-05-12T09:15:27.051 INFO:tasks.workunit.client.0.smithi028.stdout: "num_snap_trimming": 0, 2021-05-12T09:15:27.051 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shards_repaired": 0, 2021-05-12T09:15:27.051 INFO:tasks.workunit.client.0.smithi028.stdout: "op_queue_age_hist": { 2021-05-12T09:15:27.051 INFO:tasks.workunit.client.0.smithi028.stdout: "histogram": [], 2021-05-12T09:15:27.052 INFO:tasks.workunit.client.0.smithi028.stdout: "upper_bound": 1 2021-05-12T09:15:27.052 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.052 INFO:tasks.workunit.client.0.smithi028.stdout: "perf_stat": { 2021-05-12T09:15:27.052 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ms": 0, 2021-05-12T09:15:27.052 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ms": 0, 2021-05-12T09:15:27.053 INFO:tasks.workunit.client.0.smithi028.stdout: "commit_latency_ns": 0, 2021-05-12T09:15:27.053 INFO:tasks.workunit.client.0.smithi028.stdout: "apply_latency_ns": 0 2021-05-12T09:15:27.053 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.053 INFO:tasks.workunit.client.0.smithi028.stdout: "alerts": [], 2021-05-12T09:15:27.053 INFO:tasks.workunit.client.0.smithi028.stdout: "network_ping_times": [] 2021-05-12T09:15:27.054 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:27.054 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.054 INFO:tasks.workunit.client.0.smithi028.stdout: "pool_stats": [ 2021-05-12T09:15:27.054 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.054 INFO:tasks.workunit.client.0.smithi028.stdout: "poolid": 2, 2021-05-12T09:15:27.055 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 1, 2021-05-12T09:15:27.055 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:15:27.055 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 48430, 2021-05-12T09:15:27.055 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 101, 2021-05-12T09:15:27.055 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 1, 2021-05-12T09:15:27.056 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 303, 2021-05-12T09:15:27.056 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:15:27.056 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:15:27.056 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 101, 2021-05-12T09:15:27.056 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:15:27.057 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:15:27.057 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 101, 2021-05-12T09:15:27.057 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:15:27.057 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 0, 2021-05-12T09:15:27.057 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 0, 2021-05-12T09:15:27.058 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 101, 2021-05-12T09:15:27.058 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 101, 2021-05-12T09:15:27.058 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:15:27.058 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:15:27.059 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:15:27.059 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:15:27.059 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:15:27.059 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:15:27.059 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:15:27.060 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:15:27.060 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:15:27.060 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:15:27.060 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:15:27.060 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:15:27.061 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:15:27.061 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:15:27.061 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:15:27.061 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:15:27.062 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:15:27.062 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:15:27.062 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:15:27.062 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:15:27.062 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:15:27.063 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:15:27.063 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:15:27.063 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:15:27.063 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:15:27.063 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.064 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:15:27.064 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:15:27.064 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:15:27.064 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:15:27.064 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 1241088, 2021-05-12T09:15:27.065 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1241088, 2021-05-12T09:15:27.065 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:15:27.065 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:15:27.065 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:15:27.065 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:15:27.066 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:15:27.066 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.066 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 18, 2021-05-12T09:15:27.066 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 18, 2021-05-12T09:15:27.066 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 3, 2021-05-12T09:15:27.067 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 3, 2021-05-12T09:15:27.067 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 3 2021-05-12T09:15:27.067 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.067 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.068 INFO:tasks.workunit.client.0.smithi028.stdout: "poolid": 1, 2021-05-12T09:15:27.068 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg": 1, 2021-05-12T09:15:27.068 INFO:tasks.workunit.client.0.smithi028.stdout: "stat_sum": { 2021-05-12T09:15:27.068 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes": 459280, 2021-05-12T09:15:27.068 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects": 2, 2021-05-12T09:15:27.069 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_clones": 0, 2021-05-12T09:15:27.069 INFO:tasks.workunit.client.0.smithi028.stdout: "num_object_copies": 6, 2021-05-12T09:15:27.069 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing_on_primary": 0, 2021-05-12T09:15:27.069 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_missing": 0, 2021-05-12T09:15:27.069 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_degraded": 2, 2021-05-12T09:15:27.070 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_misplaced": 0, 2021-05-12T09:15:27.070 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_unfound": 0, 2021-05-12T09:15:27.070 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_dirty": 2, 2021-05-12T09:15:27.070 INFO:tasks.workunit.client.0.smithi028.stdout: "num_whiteouts": 0, 2021-05-12T09:15:27.070 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read": 76, 2021-05-12T09:15:27.071 INFO:tasks.workunit.client.0.smithi028.stdout: "num_read_kb": 64, 2021-05-12T09:15:27.071 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write": 95, 2021-05-12T09:15:27.071 INFO:tasks.workunit.client.0.smithi028.stdout: "num_write_kb": 1110, 2021-05-12T09:15:27.071 INFO:tasks.workunit.client.0.smithi028.stdout: "num_scrub_errors": 0, 2021-05-12T09:15:27.071 INFO:tasks.workunit.client.0.smithi028.stdout: "num_shallow_scrub_errors": 0, 2021-05-12T09:15:27.072 INFO:tasks.workunit.client.0.smithi028.stdout: "num_deep_scrub_errors": 0, 2021-05-12T09:15:27.072 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_recovered": 0, 2021-05-12T09:15:27.072 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_recovered": 0, 2021-05-12T09:15:27.072 INFO:tasks.workunit.client.0.smithi028.stdout: "num_keys_recovered": 0, 2021-05-12T09:15:27.072 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_omap": 0, 2021-05-12T09:15:27.073 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_hit_set_archive": 0, 2021-05-12T09:15:27.073 INFO:tasks.workunit.client.0.smithi028.stdout: "num_bytes_hit_set_archive": 0, 2021-05-12T09:15:27.073 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush": 0, 2021-05-12T09:15:27.073 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_kb": 0, 2021-05-12T09:15:27.073 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict": 0, 2021-05-12T09:15:27.074 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_kb": 0, 2021-05-12T09:15:27.074 INFO:tasks.workunit.client.0.smithi028.stdout: "num_promote": 0, 2021-05-12T09:15:27.074 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_high": 0, 2021-05-12T09:15:27.074 INFO:tasks.workunit.client.0.smithi028.stdout: "num_flush_mode_low": 0, 2021-05-12T09:15:27.075 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_some": 0, 2021-05-12T09:15:27.075 INFO:tasks.workunit.client.0.smithi028.stdout: "num_evict_mode_full": 0, 2021-05-12T09:15:27.075 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_pinned": 0, 2021-05-12T09:15:27.075 INFO:tasks.workunit.client.0.smithi028.stdout: "num_legacy_snapsets": 0, 2021-05-12T09:15:27.075 INFO:tasks.workunit.client.0.smithi028.stdout: "num_large_omap_objects": 0, 2021-05-12T09:15:27.076 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_manifest": 0, 2021-05-12T09:15:27.076 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_bytes": 0, 2021-05-12T09:15:27.076 INFO:tasks.workunit.client.0.smithi028.stdout: "num_omap_keys": 0, 2021-05-12T09:15:27.076 INFO:tasks.workunit.client.0.smithi028.stdout: "num_objects_repaired": 0 2021-05-12T09:15:27.076 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.077 INFO:tasks.workunit.client.0.smithi028.stdout: "store_stats": { 2021-05-12T09:15:27.077 INFO:tasks.workunit.client.0.smithi028.stdout: "total": 0, 2021-05-12T09:15:27.077 INFO:tasks.workunit.client.0.smithi028.stdout: "available": 0, 2021-05-12T09:15:27.077 INFO:tasks.workunit.client.0.smithi028.stdout: "internally_reserved": 0, 2021-05-12T09:15:27.077 INFO:tasks.workunit.client.0.smithi028.stdout: "allocated": 1388544, 2021-05-12T09:15:27.078 INFO:tasks.workunit.client.0.smithi028.stdout: "data_stored": 1377840, 2021-05-12T09:15:27.078 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed": 0, 2021-05-12T09:15:27.078 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_allocated": 0, 2021-05-12T09:15:27.078 INFO:tasks.workunit.client.0.smithi028.stdout: "data_compressed_original": 0, 2021-05-12T09:15:27.078 INFO:tasks.workunit.client.0.smithi028.stdout: "omap_allocated": 0, 2021-05-12T09:15:27.079 INFO:tasks.workunit.client.0.smithi028.stdout: "internal_metadata": 0 2021-05-12T09:15:27.079 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.079 INFO:tasks.workunit.client.0.smithi028.stdout: "log_size": 14, 2021-05-12T09:15:27.079 INFO:tasks.workunit.client.0.smithi028.stdout: "ondisk_log_size": 14, 2021-05-12T09:15:27.079 INFO:tasks.workunit.client.0.smithi028.stdout: "up": 2, 2021-05-12T09:15:27.080 INFO:tasks.workunit.client.0.smithi028.stdout: "acting": 2, 2021-05-12T09:15:27.080 INFO:tasks.workunit.client.0.smithi028.stdout: "num_store_stats": 3 2021-05-12T09:15:27.080 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:27.080 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:27.080 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_stats": [ 2021-05-12T09:15:27.081 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.081 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:15:27.081 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 210453397515 2021-05-12T09:15:27.081 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.081 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.082 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:15:27.082 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 223338299402 2021-05-12T09:15:27.082 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.082 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.082 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:15:27.083 INFO:tasks.workunit.client.0.smithi028.stdout: "seq": 64424509460 2021-05-12T09:15:27.083 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:27.083 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:27.083 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_by_state": [ 2021-05-12T09:15:27.083 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.084 INFO:tasks.workunit.client.0.smithi028.stdout: "state": "active+undersized+degraded", 2021-05-12T09:15:27.084 INFO:tasks.workunit.client.0.smithi028.stdout: "num": 2 2021-05-12T09:15:27.084 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:27.084 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:27.084 INFO:tasks.workunit.client.0.smithi028.stdout: "num_pg_by_osd": [ 2021-05-12T09:15:27.085 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.085 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2, 2021-05-12T09:15:27.085 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:15:27.085 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 0, 2021-05-12T09:15:27.085 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:15:27.086 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.086 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.086 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 2147483647, 2021-05-12T09:15:27.086 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:15:27.086 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 1, 2021-05-12T09:15:27.087 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:15:27.087 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.087 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.087 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 1, 2021-05-12T09:15:27.088 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 2, 2021-05-12T09:15:27.088 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 2, 2021-05-12T09:15:27.088 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:15:27.088 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.088 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.089 INFO:tasks.workunit.client.0.smithi028.stdout: "osd": 0, 2021-05-12T09:15:27.089 INFO:tasks.workunit.client.0.smithi028.stdout: "num_primary_pg": 0, 2021-05-12T09:15:27.089 INFO:tasks.workunit.client.0.smithi028.stdout: "num_acting_pg": 2, 2021-05-12T09:15:27.089 INFO:tasks.workunit.client.0.smithi028.stdout: "num_up_not_acting_pg": 0 2021-05-12T09:15:27.089 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:27.090 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:27.090 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": [ 2021-05-12T09:15:27.090 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.090 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 1, 2021-05-12T09:15:27.090 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": {} 2021-05-12T09:15:27.091 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.091 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:15:27.091 INFO:tasks.workunit.client.0.smithi028.stdout: "pool": 2, 2021-05-12T09:15:27.091 INFO:tasks.workunit.client.0.smithi028.stdout: "purged_snaps": {} 2021-05-12T09:15:27.091 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:27.092 INFO:tasks.workunit.client.0.smithi028.stdout: ], 2021-05-12T09:15:27.092 INFO:tasks.workunit.client.0.smithi028.stdout: "servicemap": { 2021-05-12T09:15:27.092 INFO:tasks.workunit.client.0.smithi028.stdout: "epoch": 5, 2021-05-12T09:15:27.092 INFO:tasks.workunit.client.0.smithi028.stdout: "modified": "2021-05-12T09:13:50.982632+0000", 2021-05-12T09:15:27.092 INFO:tasks.workunit.client.0.smithi028.stdout: "services": {} 2021-05-12T09:15:27.093 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:15:27.093 INFO:tasks.workunit.client.0.smithi028.stdout: "paxos": { 2021-05-12T09:15:27.093 INFO:tasks.workunit.client.0.smithi028.stdout: "first_committed": 1, 2021-05-12T09:15:27.093 INFO:tasks.workunit.client.0.smithi028.stdout: "last_committed": 534, 2021-05-12T09:15:27.093 INFO:tasks.workunit.client.0.smithi028.stdout: "last_pn": 0, 2021-05-12T09:15:27.094 INFO:tasks.workunit.client.0.smithi028.stdout: "accepted_pn": 0 2021-05-12T09:15:27.094 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:15:27.094 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:15:27.095 INFO:tasks.workunit.client.0.smithi028.stderr:report 1483521963 2021-05-12T09:15:27.095 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: return 1 2021-05-12T09:15:27.095 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2021-05-12T09:15:27.096 INFO:tasks.workunit.client.0.smithi028.stdout:delay recovery 2021-05-12T09:15:27.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T09:15:27.176 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:15:27.176 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-05-12T09:14:46.927659+0000 46'102 53:131 [NONE,1,0] 1 [NONE,1,0] 1 0'0 2021-05-12T09:12:53.510061+0000 0'0 2021-05-12T09:12:53.510061+0000 0 2021-05-12T09:15:27.176 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:14:46.893861+0000 32'62 53:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:12:49.062969+0000 0'0 2021-05-12T09:12:49.062969+0000 0 2021-05-12T09:15:27.177 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:15:27.177 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:15:27.178 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:15:27.187 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2021-05-12T09:15:27.188 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2021-05-12T09:15:27.188 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T09:15:27.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T09:15:27.189 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:15:27.189 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:15:27.189 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:15:27.190 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.0.asok 2021-05-12T09:15:27.190 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T09:15:27.190 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.0.asok set_recovery_delay 100000 2021-05-12T09:15:27.288 INFO:tasks.workunit.client.0.smithi028.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T09:15:27.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2021-05-12T09:15:27.299 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.1 2021-05-12T09:15:27.299 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T09:15:27.299 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T09:15:27.299 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:15:27.300 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:15:27.300 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:15:27.300 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.1.asok 2021-05-12T09:15:27.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T09:15:27.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.1.asok set_recovery_delay 100000 2021-05-12T09:15:27.399 INFO:tasks.workunit.client.0.smithi028.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T09:15:27.408 INFO:tasks.workunit.client.0.smithi028.stdout:revive divergent 2 2021-05-12T09:15:27.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 2' 2021-05-12T09:15:27.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T09:15:27.769 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:15:27.769 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-05-12T09:14:46.927659+0000 46'102 53:131 [NONE,1,0] 1 [NONE,1,0] 1 0'0 2021-05-12T09:12:53.510061+0000 0'0 2021-05-12T09:12:53.510061+0000 0 2021-05-12T09:15:27.769 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:14:46.893861+0000 32'62 53:124 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:12:49.062969+0000 0'0 2021-05-12T09:12:49.062969+0000 0 2021-05-12T09:15:27.770 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:15:27.770 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:15:27.771 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:15:27.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2021-05-12T09:15:28.384 INFO:tasks.workunit.client.0.smithi028.stderr:noup is set 2021-05-12T09:15:28.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 2 2021-05-12T09:15:28.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:15:28.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:15:28.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T09:15:28.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:15:28.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T09:15:28.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:15:28.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:15:28.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:15:28.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:15:28.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T09:15:28.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T09:15:28.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:15:28.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:15:28.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:15:28.400 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:15:28.400 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:15:28.400 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:15:28.401 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:15:28.401 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:15:28.401 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:15:28.401 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:15:28.401 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:15:28.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:15:28.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:15:28.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:15:28.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:15:28.403 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:15:28.403 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:15:28.403 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:15:28.403 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:15:28.403 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T09:15:28.404 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:15:28.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T09:15:28.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:15:28.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T09:15:28.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T09:15:28.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:15:28.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:15:28.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:15:28.425 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:15:28.423+0000 7f4b2ec3a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:15:28.428 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:15:28.425+0000 7f4b2ec3a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:15:28.432 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:15:28.426+0000 7f4b2ec3a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:15:28.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: return 0 2021-05-12T09:15:28.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2021-05-12T09:15:28.945 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:15:28.944+0000 7f4b2ec3a0c0 -1 Falling back to public interface 2021-05-12T09:15:31.318 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:15:31.316+0000 7f4b2ec3a0c0 -1 osd.2 33 log_to_monitors {default=true} 2021-05-12T09:15:33.789 INFO:tasks.workunit.client.0.smithi028.stdout:delay recovery divergent 2021-05-12T09:15:33.790 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2021-05-12T09:15:33.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T09:15:34.164 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:15:34.165 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-05-12T09:14:46.927659+0000 46'102 54:133 [NONE,1,0] 1 [NONE,1,0] 1 0'0 2021-05-12T09:12:53.510061+0000 0'0 2021-05-12T09:12:53.510061+0000 0 2021-05-12T09:15:34.165 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:14:46.893861+0000 32'62 54:126 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:12:49.062969+0000 0'0 2021-05-12T09:12:49.062969+0000 0 2021-05-12T09:15:34.165 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:15:34.166 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:15:34.167 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:15:34.177 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.2 2021-05-12T09:15:34.177 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-05-12T09:15:34.178 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-05-12T09:15:34.178 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:15:34.179 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:15:34.179 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:15:34.179 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29719/ceph-osd.2.asok 2021-05-12T09:15:34.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2021-05-12T09:15:34.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.29719/ceph-osd.2.asok set_recovery_delay 100000 2021-05-12T09:15:34.278 INFO:tasks.workunit.client.0.smithi028.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-05-12T09:15:34.287 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2021-05-12T09:15:35.116 INFO:tasks.workunit.client.0.smithi028.stderr:noup is unset 2021-05-12T09:15:35.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2021-05-12T09:15:35.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:15:35.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:15:35.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:15:35.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:15:35.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:15:35.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:15:35.128 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:15:35.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:15:35.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:15:35.598 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 49 up_thru 51 down_at 48 last_clean_interval [40,46) [v2:127.0.0.1:6802/79547,v1:127.0.0.1:6803/79547] [v2:127.0.0.1:6804/79547,v1:127.0.0.1:6805/79547] exists,up 211d7d1b-7ece-4928-932a-3aa488ac19a5 2021-05-12T09:15:35.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:15:35.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:15:35.600 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:15:35.600 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2021-05-12T09:15:35.600 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:15:35.600 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:15:35.601 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:15:35.601 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:15:35.601 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:15:35.601 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:15:35.602 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:15:35.602 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:15:35.602 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:15:35.981 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 52 up_thru 58 down_at 51 last_clean_interval [42,46) [v2:127.0.0.1:6810/80230,v1:127.0.0.1:6811/80230] [v2:127.0.0.1:6812/80230,v1:127.0.0.1:6813/80230] exists,up 860f4a3c-1b2f-41ed-b1b5-8d8eaa20b04b 2021-05-12T09:15:35.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:15:35.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:15:35.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:15:35.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2021-05-12T09:15:35.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:15:35.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:15:35.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:15:35.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:15:35.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:15:35.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:15:35.985 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:15:35.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:15:35.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:15:36.517 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 58 up_thru 58 down_at 36 last_clean_interval [15,33) [v2:127.0.0.1:6818/82398,v1:127.0.0.1:6819/82398] [v2:127.0.0.1:6820/82398,v1:127.0.0.1:6821/82398] exists,up 7951a78e-ee8e-4c4f-a30b-05cc781094fa 2021-05-12T09:15:36.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:15:36.519 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:15:36.519 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:15:36.519 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T09:15:36.972 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:15:36.972 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-05-12T09:14:46.927659+0000 46'102 54:133 [NONE,1,0] 1 [NONE,1,0] 1 0'0 2021-05-12T09:12:53.510061+0000 0'0 2021-05-12T09:12:53.510061+0000 0 2021-05-12T09:15:36.973 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 2 0 0 459280 0 0 14 14 active+undersized+degraded 2021-05-12T09:14:46.893861+0000 32'62 54:126 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:12:49.062969+0000 0'0 2021-05-12T09:12:49.062969+0000 0 2021-05-12T09:15:36.973 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:15:36.973 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:15:36.974 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:15:36.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2021-05-12T09:15:36.983 INFO:tasks.workunit.client.0.smithi028.stdout:wait for peering 2021-05-12T09:15:36.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T09:15:37.350 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:15:37.350 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2021-05-12T09:15:35.414434+0000 46'102 59:131 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:12:53.510061+0000 0'0 2021-05-12T09:12:53.510061+0000 0 2021-05-12T09:15:37.350 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:15:35.589493+0000 32'62 60:140 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:12:49.062969+0000 0'0 2021-05-12T09:12:49.062969+0000 0 2021-05-12T09:15:37.351 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:15:37.351 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:15:37.351 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:15:37.361 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2021-05-12T09:15:42.995 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 2' 2021-05-12T09:15:42.996 INFO:tasks.workunit.client.0.smithi028.stdout:killing divergent 2 2021-05-12T09:15:42.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T09:15:43.378 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:15:43.379 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2021-05-12T09:15:35.414434+0000 46'102 59:131 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:12:53.510061+0000 0'0 2021-05-12T09:12:53.510061+0000 0 2021-05-12T09:15:43.379 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:15:35.589493+0000 32'62 60:140 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:12:49.062969+0000 0'0 2021-05-12T09:12:49.062969+0000 0 2021-05-12T09:15:43.379 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:15:43.379 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:15:43.380 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:15:43.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2021-05-12T09:15:43.390 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:15:43.390 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:15:43.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:15:43.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:15:43.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:15:43.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:15:43.496 INFO:tasks.workunit.client.0.smithi028.stdout:reviving divergent 2 2021-05-12T09:15:43.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 2' 2021-05-12T09:15:43.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T09:15:43.861 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:15:43.862 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2021-05-12T09:15:35.414434+0000 46'102 59:131 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:12:53.510061+0000 0'0 2021-05-12T09:12:53.510061+0000 0 2021-05-12T09:15:43.862 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:15:35.589493+0000 32'62 60:140 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:12:49.062969+0000 0'0 2021-05-12T09:12:49.062969+0000 0 2021-05-12T09:15:43.862 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:15:43.863 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:15:43.863 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:15:43.876 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 2 2021-05-12T09:15:43.876 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/divergent-priors 2021-05-12T09:15:43.876 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:15:43.876 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T09:15:43.877 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:15:43.877 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/divergent-priors/2 2021-05-12T09:15:43.877 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-05-12T09:15:43.877 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:15:43.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:15:43.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:15:43.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-05-12T09:15:43.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-05-12T09:15:43.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:15:43.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:15:43.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-05-12T09:15:43.879 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:15:43.880 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:15:43.880 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:15:43.880 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:15:43.880 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:15:43.881 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:15:43.881 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:15:43.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' 2021-05-12T09:15:43.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:15:43.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-05-12T09:15:43.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-05-12T09:15:43.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:15:43.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:15:43.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:15:43.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:15:43.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:15:43.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/divergent-priors/2 2021-05-12T09:15:43.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T09:15:43.884 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:15:43.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=8342ec9e-531f-44b4-8c8c-079d667fe908 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.29719/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:15:43.885 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/divergent-priors/2/whoami 2021-05-12T09:15:43.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T09:15:43.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:15:43.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:15:43.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:15:43.904 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:15:43.903+0000 7f3f20a010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:15:43.906 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:15:43.904+0000 7f3f20a010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:15:43.907 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:15:43.906+0000 7f3f20a010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:15:44.268 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T09:15:44.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:15:44.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:15:44.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:15:44.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:15:44.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:15:44.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:15:44.270 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:15:44.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:15:44.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:15:44.425 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:15:44.423+0000 7f3f20a010c0 -1 Falling back to public interface 2021-05-12T09:15:44.657 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 58 up_thru 58 down_at 36 last_clean_interval [15,33) [v2:127.0.0.1:6818/82398,v1:127.0.0.1:6819/82398] [v2:127.0.0.1:6820/82398,v1:127.0.0.1:6821/82398] exists,up 7951a78e-ee8e-4c4f-a30b-05cc781094fa 2021-05-12T09:15:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:15:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:15:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:15:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2021-05-12T09:15:46.871 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:15:46.870+0000 7f3f20a010c0 -1 osd.2 60 log_to_monitors {default=true} 2021-05-12T09:16:04.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2021-05-12T09:16:04.659 INFO:tasks.workunit.client.0.smithi028.stdout:allowing recovery 2021-05-12T09:16:04.659 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T09:16:05.036 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:16:05.037 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2021-05-12T09:15:55.150765+0000 60'103 64:155 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:12:53.510061+0000 0'0 2021-05-12T09:12:53.510061+0000 0 2021-05-12T09:16:05.037 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:15:48.134552+0000 32'62 64:153 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:12:49.062969+0000 0'0 2021-05-12T09:12:49.062969+0000 0 2021-05-12T09:16:05.038 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:16:05.038 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:16:05.039 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:16:05.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-05-12T09:16:05.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2021-05-12T09:16:05.179 INFO:tasks.workunit.client.0.smithi028.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T09:16:05.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-05-12T09:16:05.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2021-05-12T09:16:05.319 INFO:tasks.workunit.client.0.smithi028.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T09:16:05.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-05-12T09:16:05.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2021-05-12T09:16:05.460 INFO:tasks.workunit.client.0.smithi028.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-05-12T09:16:05.469 INFO:tasks.workunit.client.0.smithi028.stdout:reading divergent objects 2021-05-12T09:16:05.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2021-05-12T09:16:05.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2021-05-12T09:16:05.841 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:16:05.841 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2021-05-12T09:15:55.150765+0000 60'103 64:155 [2,1,0] 2 [2,1,0] 2 0'0 2021-05-12T09:12:53.510061+0000 0'0 2021-05-12T09:12:53.510061+0000 0 2021-05-12T09:16:05.841 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 2 0 0 0 0 459280 0 0 14 14 active+clean 2021-05-12T09:15:48.134552+0000 32'62 64:153 [1,0,2] 1 [1,0,2] 1 0'0 2021-05-12T09:12:49.062969+0000 0'0 2021-05-12T09:12:49.062969+0000 0 2021-05-12T09:16:05.842 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:16:05.842 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:16:05.843 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:16:05.853 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2021-05-12T09:16:05.854 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2021-05-12T09:16:05.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:16:05.857 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2021-05-12T09:16:05.899 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:16:05.899 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2021-05-12T09:16:05.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:16:05.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2021-05-12T09:16:05.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-05-12T09:16:05.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2021-05-12T09:16:05.993 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2021-05-12T09:16:05.994 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2021-05-12T09:16:05.995 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T09:16:06.000 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 20 _merge_object_divergent_entries: merging hoid 2:7a80c0d5:::existing_99:1 entries: 33'103 (32'99) clone 2:7a80c0d5:::existing_99:1 by unknown.0.0:0 2021-05-12T09:13:28.104135+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T09:16:06.001 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 20 _merge_object_divergent_entries: keeping 33'103 (32'99) clone 2:7a80c0d5:::existing_99:1 by unknown.0.0:0 2021-05-12T09:13:28.104135+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-05-12T09:16:06.001 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T09:16:06.001 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:7a80c0d5:::existing_99:1 prior_version: 32'99 first_divergent_update: 33'103 last_divergent_update: 33'103 2021-05-12T09:16:06.001 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:7a80c0d5:::existing_99:1 has no more recent entries in log 2021-05-12T09:16:06.002 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2021-05-12T09:16:06.002 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 20 _merge_object_divergent_entries: merging hoid 2:7a80c0d5:::existing_99:head entries: 33'104 (32'99) modify 2:7a80c0d5:::existing_99:head by client.4563.0:1 2021-05-12T09:13:35.597084+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:16:06.002 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 20 _merge_object_divergent_entries: keeping 33'104 (32'99) modify 2:7a80c0d5:::existing_99:head by client.4563.0:1 2021-05-12T09:13:35.597084+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:16:06.002 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T09:16:06.002 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:7a80c0d5:::existing_99:head prior_version: 32'99 first_divergent_update: 33'104 last_divergent_update: 33'104 2021-05-12T09:16:06.003 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:7a80c0d5:::existing_99:head has no more recent entries in log 2021-05-12T09:16:06.003 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2021-05-12T09:16:06.003 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:7a80c0d5:::existing_99:head olog_can_rollback_to: 33'104 2021-05-12T09:16:06.003 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:7a80c0d5:::existing_99:head cannot rollback 33'104 (32'99) modify 2:7a80c0d5:::existing_99:head by client.4563.0:1 2021-05-12T09:13:35.597084+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:16:06.004 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2021-05-12T09:16:06.004 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 20 _merge_object_divergent_entries: merging hoid 2:c156033e:::existing_100:head entries: 32'101 (32'100) modify 2:c156033e:::existing_100:head by client.4554.0:1 2021-05-12T09:13:34.388610+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,32'102 (32'101) modify 2:c156033e:::existing_100:head by client.4560.0:1 2021-05-12T09:13:35.395235+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:16:06.004 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 20 _merge_object_divergent_entries: keeping 32'101 (32'100) modify 2:c156033e:::existing_100:head by client.4554.0:1 2021-05-12T09:13:34.388610+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:16:06.005 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 20 _merge_object_divergent_entries: keeping 32'102 (32'101) modify 2:c156033e:::existing_100:head by client.4560.0:1 2021-05-12T09:13:35.395235+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:16:06.005 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-05-12T09:16:06.005 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:c156033e:::existing_100:head prior_version: 32'100 first_divergent_update: 32'101 last_divergent_update: 32'102 2021-05-12T09:16:06.005 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:c156033e:::existing_100:head has no more recent entries in log 2021-05-12T09:16:06.006 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2021-05-12T09:16:06.006 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:c156033e:::existing_100:head olog_can_rollback_to: 33'104 2021-05-12T09:16:06.006 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:c156033e:::existing_100:head cannot rollback 32'102 (32'101) modify 2:c156033e:::existing_100:head by client.4560.0:1 2021-05-12T09:13:35.395235+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-05-12T09:16:06.006 INFO:tasks.workunit.client.0.smithi028.stdout:td/divergent-priors/osd.2.log:2021-05-12T09:15:35.079+0000 7f4b0ab2f700 10 _merge_object_divergent_entries: hoid 2:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2021-05-12T09:16:06.007 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2021-05-12T09:16:06.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log 2021-05-12T09:16:06.010 INFO:tasks.workunit.client.0.smithi028.stdout:success 2021-05-12T09:16:06.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2021-05-12T09:16:06.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2021-05-12T09:16:06.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-05-12T09:16:06.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-05-12T09:16:06.554 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'test' does not exist 2021-05-12T09:16:06.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2021-05-12T09:16:06.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:16:06.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:16:06.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:16:06.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:16:06.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:16:06.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:16:06.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-05-12T09:16:06.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T09:16:06.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:16:06.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T09:16:06.674 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:16:06.674 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:16:06.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:16:06.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:16:06.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:16:06.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:16:06.677 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:16:06.678 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:16:06.678 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:16:06.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:16:06.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:16:06.680 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:16:06.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:16:06.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:16:06.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:16:06.682 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:16:06.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:16:06.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:16:06.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T09:16:06.699 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:16:06.699 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:06.699 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:16:06.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29719 2021-05-12T09:16:06.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:16:06.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:16:06.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T09:16:06.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/divergent-priors 0 2021-05-12T09:16:06.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-05-12T09:16:06.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T09:16:06.703 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-05-12T09:16:06.703 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:16:06.703 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:16:06.703 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:16:06.703 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:16:06.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:16:06.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:16:06.705 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:16:06.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:16:06.705 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:16:06.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:16:06.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:16:06.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:16:06.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:16:06.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:16:06.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:16:06.708 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:16:06.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:16:06.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T09:16:06.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-05-12T09:16:06.710 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:16:06.710 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:06.710 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29719 2021-05-12T09:16:06.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29719 2021-05-12T09:16:06.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:16:06.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:16:06.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T09:16:06.713 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T09:16:06.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T09:16:06.788 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2021-05-12T09:16:06.789 DEBUG:teuthology.orchestra.run.smithi028:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2021-05-12T09:16:06.849 INFO:tasks.workunit.client.0.smithi028.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T09:16:06.857 INFO:tasks.workunit.client.0.smithi028.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T09:16:06.858 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:16:06.858 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:16:06.858 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T09:16:06.858 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T09:16:06.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T09:16:06.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T09:16:06.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T09:16:06.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T09:16:06.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/ec-error-rollforward 2021-05-12T09:16:06.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2021-05-12T09:16:06.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2021-05-12T09:16:06.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2021-05-12T09:16:06.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2021-05-12T09:16:06.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2021-05-12T09:16:06.861 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2021-05-12T09:16:06.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none ' 2021-05-12T09:16:06.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2021-05-12T09:16:06.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2021-05-12T09:16:06.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2021-05-12T09:16:06.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2021-05-12T09:16:06.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2021-05-12T09:16:06.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2021-05-12T09:16:06.863 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2021-05-12T09:16:06.863 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2021-05-12T09:16:06.863 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T09:16:06.866 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2021-05-12T09:16:06.866 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2021-05-12T09:16:06.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2021-05-12T09:16:06.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2021-05-12T09:16:06.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2021-05-12T09:16:06.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-05-12T09:16:06.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:16:06.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-05-12T09:16:06.868 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:16:06.868 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:16:06.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:16:06.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:16:06.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:16:06.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:16:06.869 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:16:06.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:16:06.870 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:16:06.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:16:06.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:16:06.870 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:16:06.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:16:06.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:16:06.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:16:06.871 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:16:06.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:16:06.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:16:06.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-05-12T09:16:06.875 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:16:06.875 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:06.875 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:16:06.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83966 2021-05-12T09:16:06.876 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:16:06.876 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:16:06.876 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2021-05-12T09:16:06.876 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:16:06.877 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:06.877 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:16:06.877 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83966 2021-05-12T09:16:06.877 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:16:06.877 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T09:16:06.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T09:16:06.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:16:06.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2021-05-12T09:16:06.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2021-05-12T09:16:06.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2021-05-12T09:16:06.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2021-05-12T09:16:06.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2021-05-12T09:16:06.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:16:06.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:16:06.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:16:06.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2021-05-12T09:16:06.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2021-05-12T09:16:07.240 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:16:07.240 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:16:07.240 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:16:07.241 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:16:07.241 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:07.241 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:16:07.241 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:07.242 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:16:07.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:16:07.477 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:16:07.477 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:16:07.477 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:16:07.477 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:16:07.478 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:16:07.478 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:16:07.478 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:16:07.478 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:16:07.479 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:16:07.479 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:07.479 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:16:07.479 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83966/ceph-mon.a.asok 2021-05-12T09:16:07.479 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:16:07.480 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.83966/ceph-mon.a.asok config get fsid 2021-05-12T09:16:07.576 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:16:07.577 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:16:07.577 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:16:07.577 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:16:07.578 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:16:07.578 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:16:07.578 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:16:07.578 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:16:07.578 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:16:07.579 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:07.579 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:16:07.579 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83966/ceph-mon.a.asok 2021-05-12T09:16:07.579 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:16:07.580 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.83966/ceph-mon.a.asok config get mon_host 2021-05-12T09:16:07.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2021-05-12T09:16:07.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2021-05-12T09:16:07.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:16:07.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:16:07.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:16:07.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2021-05-12T09:16:07.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:16:08.556 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:16:08.556 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:16:08.557 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:16:08.557 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:16:08.557 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:08.557 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:16:08.557 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:08.558 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:16:08.558 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:16:08.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2021-05-12T09:16:08.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-05-12T09:16:08.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:16:08.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:16:08.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:16:08.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2021-05-12T09:16:08.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T09:16:08.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:16:08.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:16:08.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:16:08.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2021-05-12T09:16:08.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2021-05-12T09:16:08.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:16:08.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:16:08.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T09:16:08.585 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:16:08.586 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:16:08.586 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:16:08.586 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:16:08.586 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:08.587 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:16:08.587 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:08.587 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:08.587 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:16:08.587 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:16:08.588 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:16:08.588 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T09:16:08.588 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T09:16:08.588 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:16:08.588 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:16:08.589 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:08.589 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:16:08.589 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:16:08.589 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/ec-error-rollforward/0 2021-05-12T09:16:08.590 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:16:08.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=3deb75ed-0f1d-43ea-8c7b-1e598fbdc8ee 2021-05-12T09:16:08.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 3deb75ed-0f1d-43ea-8c7b-1e598fbdc8ee' 2021-05-12T09:16:08.591 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 3deb75ed-0f1d-43ea-8c7b-1e598fbdc8ee 2021-05-12T09:16:08.592 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:16:08.595 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDYnJtgoBFfIxAADbDR3ax5W6Qq/M1Crl4e5Q== 2021-05-12T09:16:08.595 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDYnJtgoBFfIxAADbDR3ax5W6Qq/M1Crl4e5Q=="}' 2021-05-12T09:16:08.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 3deb75ed-0f1d-43ea-8c7b-1e598fbdc8ee -i td/ec-error-rollforward/0/new.json 2021-05-12T09:16:08.917 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:16:08.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/ec-error-rollforward/0/new.json 2021-05-12T09:16:08.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDYnJtgoBFfIxAADbDR3ax5W6Qq/M1Crl4e5Q== --osd-uuid 3deb75ed-0f1d-43ea-8c7b-1e598fbdc8ee 2021-05-12T09:16:08.951 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:08.949+0000 7fa7e766c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:08.953 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:08.952+0000 7fa7e766c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:08.955 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:08.954+0000 7fa7e766c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:08.956 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:08.954+0000 7fa7e766c0c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-05-12T09:16:08.956 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:08.954+0000 7fa7e766c0c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-05-12T09:16:08.957 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:08.954+0000 7fa7e766c0c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-05-12T09:16:08.989 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:08.988+0000 7fa7e766c0c0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2021-05-12T09:16:13.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2021-05-12T09:16:13.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:16:13.278 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:16:13.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:16:13.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:16:13.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:16:13.530 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:16:13.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:13.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:16:13.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:16:13.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:16:13.550 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:13.549+0000 7fbf5e7aa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:13.552 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:13.551+0000 7fbf5e7aa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:13.553 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:13.552+0000 7fbf5e7aa0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:13.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:16:13.908 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:16:13.908 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:16:13.908 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:16:13.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:16:13.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:13.909 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:16:13.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:16:13.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:13.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:16:14.072 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:14.071+0000 7fbf5e7aa0c0 -1 Falling back to public interface 2021-05-12T09:16:14.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:16:15.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:16:15.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:15.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:16:15.283 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:16:15.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:15.284 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:16:15.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:16:16.315 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:16.313+0000 7fbf5e7aa0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:16:16.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:16:16.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:16.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:16:16.693 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:16:16.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:16.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:16:17.079 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85086,v1:127.0.0.1:6803/85086] [v2:127.0.0.1:6804/85086,v1:127.0.0.1:6805/85086] exists,up 3deb75ed-0f1d-43ea-8c7b-1e598fbdc8ee 2021-05-12T09:16:17.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:16:17.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:16:17.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:16:17.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2021-05-12T09:16:17.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-05-12T09:16:17.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:16:17.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:16:17.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:16:17.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2021-05-12T09:16:17.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T09:16:17.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:16:17.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:16:17.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:16:17.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2021-05-12T09:16:17.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2021-05-12T09:16:17.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:16:17.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:16:17.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T09:16:17.088 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:16:17.088 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:16:17.088 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:16:17.088 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:16:17.088 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:17.089 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:16:17.089 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:17.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:17.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:16:17.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:16:17.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:16:17.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T09:16:17.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T09:16:17.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:16:17.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:16:17.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:17.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:16:17.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:16:17.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/ec-error-rollforward/1 2021-05-12T09:16:17.092 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:16:17.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=103e5b12-c0eb-48ea-93b8-2e63fc158391 2021-05-12T09:16:17.092 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 103e5b12-c0eb-48ea-93b8-2e63fc158391 2021-05-12T09:16:17.093 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 103e5b12-c0eb-48ea-93b8-2e63fc158391' 2021-05-12T09:16:17.093 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:16:17.095 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDhnJtgDd+RBRAAKFS7dFEqlW1OYTapGgU4qg== 2021-05-12T09:16:17.095 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDhnJtgDd+RBRAAKFS7dFEqlW1OYTapGgU4qg=="}' 2021-05-12T09:16:17.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 103e5b12-c0eb-48ea-93b8-2e63fc158391 -i td/ec-error-rollforward/1/new.json 2021-05-12T09:16:17.665 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:16:17.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/ec-error-rollforward/1/new.json 2021-05-12T09:16:17.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDhnJtgDd+RBRAAKFS7dFEqlW1OYTapGgU4qg== --osd-uuid 103e5b12-c0eb-48ea-93b8-2e63fc158391 2021-05-12T09:16:17.698 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:17.696+0000 7fa8791750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:17.700 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:17.698+0000 7fa8791750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:17.702 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:17.700+0000 7fa8791750c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:17.702 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:17.701+0000 7fa8791750c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-05-12T09:16:17.703 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:17.701+0000 7fa8791750c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-05-12T09:16:17.703 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:17.701+0000 7fa8791750c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-05-12T09:16:17.751 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:17.750+0000 7fa8791750c0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2021-05-12T09:16:22.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2021-05-12T09:16:22.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:16:22.026 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:16:22.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:16:22.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:16:22.544 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:16:22.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:16:22.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:22.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:16:22.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:16:22.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:16:22.564 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:22.563+0000 7ff9a97fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:22.592 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:22.590+0000 7ff9a97fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:22.593 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:22.591+0000 7ff9a97fd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:22.924 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:16:22.924 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:16:22.924 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:16:22.925 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:16:22.925 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:16:22.925 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:22.925 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:16:22.926 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:16:22.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:22.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:16:23.109 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:23.107+0000 7ff9a97fd0c0 -1 Falling back to public interface 2021-05-12T09:16:23.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:16:24.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:16:24.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:24.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:16:24.328 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:16:24.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:24.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:16:24.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:16:25.337 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:25.334+0000 7ff9a97fd0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:16:25.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:16:25.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:25.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:16:25.720 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:16:25.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:25.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:16:26.106 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/86471,v1:127.0.0.1:6811/86471] [v2:127.0.0.1:6812/86471,v1:127.0.0.1:6813/86471] exists,up 103e5b12-c0eb-48ea-93b8-2e63fc158391 2021-05-12T09:16:26.111 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:16:26.111 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:16:26.111 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:16:26.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2021-05-12T09:16:26.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-05-12T09:16:26.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:16:26.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T09:16:26.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:16:26.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2021-05-12T09:16:26.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T09:16:26.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:16:26.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:16:26.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:16:26.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2021-05-12T09:16:26.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2021-05-12T09:16:26.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:16:26.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:16:26.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T09:16:26.115 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:16:26.115 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:16:26.115 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:16:26.115 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:16:26.116 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:26.116 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:16:26.116 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:26.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:26.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:16:26.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:16:26.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:16:26.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T09:16:26.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T09:16:26.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:16:26.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:16:26.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:26.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:16:26.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:16:26.119 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/ec-error-rollforward/2 2021-05-12T09:16:26.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:16:26.119 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=2242a79b-33a7-43a0-a01a-80c852755063 2021-05-12T09:16:26.120 INFO:tasks.workunit.client.0.smithi028.stdout:add osd2 2242a79b-33a7-43a0-a01a-80c852755063 2021-05-12T09:16:26.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 2242a79b-33a7-43a0-a01a-80c852755063' 2021-05-12T09:16:26.120 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:16:26.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDqnJtgy/9LBxAA68a5/w3oIJFRS+ncc0pqEA== 2021-05-12T09:16:26.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDqnJtgy/9LBxAA68a5/w3oIJFRS+ncc0pqEA=="}' 2021-05-12T09:16:26.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 2242a79b-33a7-43a0-a01a-80c852755063 -i td/ec-error-rollforward/2/new.json 2021-05-12T09:16:26.635 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:16:26.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/ec-error-rollforward/2/new.json 2021-05-12T09:16:26.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDqnJtgy/9LBxAA68a5/w3oIJFRS+ncc0pqEA== --osd-uuid 2242a79b-33a7-43a0-a01a-80c852755063 2021-05-12T09:16:26.668 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:26.666+0000 7fcc6033c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:26.670 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:26.668+0000 7fcc6033c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:26.672 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:26.669+0000 7fcc6033c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:26.673 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:26.670+0000 7fcc6033c0c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-05-12T09:16:26.673 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:26.670+0000 7fcc6033c0c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-05-12T09:16:26.673 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:26.670+0000 7fcc6033c0c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-05-12T09:16:26.706 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:26.703+0000 7fcc6033c0c0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2021-05-12T09:16:31.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2021-05-12T09:16:31.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:16:31.472 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd2 key to auth repository 2021-05-12T09:16:31.473 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T09:16:31.473 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:16:31.981 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:16:31.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T09:16:31.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:31.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:16:31.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:16:31.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:16:32.001 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:31.998+0000 7fe0c428c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:32.003 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:32.000+0000 7fe0c428c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:32.004 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:32.001+0000 7fe0c428c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:32.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T09:16:32.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:16:32.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:16:32.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:16:32.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:16:32.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:32.358 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:16:32.358 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:16:32.358 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:32.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:16:32.525 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:32.522+0000 7fe0c428c0c0 -1 Falling back to public interface 2021-05-12T09:16:32.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:16:33.753 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:16:33.753 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:33.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:16:33.754 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:16:33.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:33.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:16:34.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:16:34.744 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:34.741+0000 7fe0c428c0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T09:16:35.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:16:35.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:35.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:16:35.146 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:16:35.146 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:35.147 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:16:35.596 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/87856,v1:127.0.0.1:6819/87856] [v2:127.0.0.1:6820/87856,v1:127.0.0.1:6821/87856] exists,up 2242a79b-33a7-43a0-a01a-80c852755063 2021-05-12T09:16:35.600 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:16:35.600 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:16:35.600 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:16:35.600 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2021-05-12T09:16:35.601 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-05-12T09:16:35.601 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:16:35.601 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-05-12T09:16:35.601 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:16:35.602 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2021-05-12T09:16:35.602 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T09:16:35.602 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:16:35.603 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:16:35.603 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:16:35.603 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2021-05-12T09:16:35.603 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2021-05-12T09:16:35.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:16:35.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:16:35.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T09:16:35.604 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:16:35.604 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:16:35.605 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:16:35.605 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:16:35.605 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:35.606 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:16:35.606 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:35.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:35.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:16:35.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:16:35.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:16:35.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T09:16:35.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T09:16:35.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:16:35.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:16:35.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:35.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:16:35.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:16:35.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/ec-error-rollforward/3 2021-05-12T09:16:35.609 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:16:35.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=43f7f278-823e-4a0b-972d-59bbacc89ce8 2021-05-12T09:16:35.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 43f7f278-823e-4a0b-972d-59bbacc89ce8' 2021-05-12T09:16:35.610 INFO:tasks.workunit.client.0.smithi028.stdout:add osd3 43f7f278-823e-4a0b-972d-59bbacc89ce8 2021-05-12T09:16:35.611 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:16:35.614 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDznJtgtqJ9JBAAq7MXMfMUVBYqRPclp+AwOQ== 2021-05-12T09:16:35.614 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDznJtgtqJ9JBAAq7MXMfMUVBYqRPclp+AwOQ=="}' 2021-05-12T09:16:35.614 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 43f7f278-823e-4a0b-972d-59bbacc89ce8 -i td/ec-error-rollforward/3/new.json 2021-05-12T09:16:36.286 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:16:36.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/ec-error-rollforward/3/new.json 2021-05-12T09:16:36.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDznJtgtqJ9JBAAq7MXMfMUVBYqRPclp+AwOQ== --osd-uuid 43f7f278-823e-4a0b-972d-59bbacc89ce8 2021-05-12T09:16:36.319 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:36.316+0000 7f385c3e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:36.321 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:36.318+0000 7f385c3e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:36.322 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:36.320+0000 7f385c3e60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:36.323 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:36.321+0000 7f385c3e60c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-05-12T09:16:36.323 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:36.321+0000 7f385c3e60c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-05-12T09:16:36.324 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:36.321+0000 7f385c3e60c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-05-12T09:16:36.407 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:36.404+0000 7f385c3e60c0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2021-05-12T09:16:43.430 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2021-05-12T09:16:43.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:16:43.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2021-05-12T09:16:43.432 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd3 key to auth repository 2021-05-12T09:16:43.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:16:43.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2021-05-12T09:16:43.946 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.3 2021-05-12T09:16:43.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:43.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:16:43.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:16:43.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:16:43.965 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:43.964+0000 7f07c38a50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:43.967 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:43.966+0000 7f07c38a50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:43.968 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:43.967+0000 7f07c38a50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:44.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2021-05-12T09:16:44.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:16:44.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T09:16:44.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:16:44.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:16:44.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:44.326 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:16:44.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:16:44.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:44.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:16:44.485 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:44.484+0000 7f07c38a50c0 -1 Falling back to public interface 2021-05-12T09:16:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:16:45.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:16:45.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:45.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:16:45.707 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:16:45.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:45.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:16:46.093 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:16:46.631 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:46.629+0000 7f07c38a50c0 -1 osd.3 0 log_to_monitors {default=true} 2021-05-12T09:16:47.095 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:16:47.095 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:47.095 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:16:47.096 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:16:47.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:47.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:16:47.529 INFO:tasks.workunit.client.0.smithi028.stdout:osd.3 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/89244,v1:127.0.0.1:6827/89244] [v2:127.0.0.1:6828/89244,v1:127.0.0.1:6829/89244] exists,up 43f7f278-823e-4a0b-972d-59bbacc89ce8 2021-05-12T09:16:47.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:16:47.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:16:47.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:16:47.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2021-05-12T09:16:48.181 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2021-05-12T09:16:48.899 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'ec' already exists 2021-05-12T09:16:48.908 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2021-05-12T09:16:50.079 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2021-05-12T09:16:50.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 87856 2021-05-12T09:16:50.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=89856 2021-05-12T09:16:50.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2021-05-12T09:16:50.088 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2021-05-12T09:16:51.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 89877' 2021-05-12T09:16:51.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2021-05-12T09:16:51.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 89878' 2021-05-12T09:16:51.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2021-05-12T09:16:51.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 89879' 2021-05-12T09:16:51.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2021-05-12T09:16:51.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2021-05-12T09:16:52.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2021-05-12T09:16:52.091 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:16:52.092 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:16:52.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:16:52.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:16:52.093 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:16:52.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:16:52.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 89856 89877 89878 89879 2021-05-12T09:16:52.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2021-05-12T09:16:52.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2021-05-12T09:16:52.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/ec-error-rollforward 2021-05-12T09:16:52.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:16:52.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T09:16:52.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:16:52.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/ec-error-rollforward/0 2021-05-12T09:16:52.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T09:16:52.206 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:16:52.206 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:16:52.206 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:16:52.206 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2021-05-12T09:16:52.207 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2021-05-12T09:16:52.207 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:16:52.207 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:16:52.207 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T09:16:52.208 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:16:52.208 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:16:52.208 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:16:52.208 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:16:52.210 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:52.211 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:16:52.211 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:52.211 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:52.211 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:16:52.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T09:16:52.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T09:16:52.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:16:52.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:16:52.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:52.213 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:16:52.214 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:16:52.214 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/ec-error-rollforward/0 2021-05-12T09:16:52.214 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T09:16:52.215 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:16:52.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:52.216 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/ec-error-rollforward/0/whoami 2021-05-12T09:16:52.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T09:16:52.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:16:52.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:16:52.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:16:52.228 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:52.226+0000 7fd71cfcb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:52.230 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:52.228+0000 7fd71cfcb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:52.233 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:52.231+0000 7fd71cfcb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:52.594 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T09:16:52.594 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:16:52.595 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:16:52.595 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:16:52.595 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:16:52.595 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:52.595 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:16:52.596 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:16:52.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:52.597 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:16:52.749 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:52.747+0000 7fd71cfcb0c0 -1 Falling back to public interface 2021-05-12T09:16:52.994 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85086,v1:127.0.0.1:6803/85086] [v2:127.0.0.1:6804/85086,v1:127.0.0.1:6805/85086] exists,up 3deb75ed-0f1d-43ea-8c7b-1e598fbdc8ee 2021-05-12T09:16:52.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:16:52.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:16:52.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:16:52.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2021-05-12T09:16:52.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/ec-error-rollforward 2021-05-12T09:16:52.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:16:52.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T09:16:52.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:16:52.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/ec-error-rollforward/1 2021-05-12T09:16:52.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T09:16:53.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:16:53.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:16:53.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:16:53.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2021-05-12T09:16:53.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2021-05-12T09:16:53.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:16:53.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:16:53.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T09:16:53.001 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:16:53.002 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:16:53.002 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:16:53.002 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:16:53.002 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:53.002 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:16:53.003 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:53.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:53.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:16:53.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T09:16:53.004 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T09:16:53.004 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:16:53.004 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:16:53.004 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:53.004 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:16:53.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:16:53.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/ec-error-rollforward/1 2021-05-12T09:16:53.005 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:16:53.006 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T09:16:53.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:53.007 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/ec-error-rollforward/1/whoami 2021-05-12T09:16:53.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T09:16:53.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:16:53.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:16:53.008 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:16:53.015 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:53.014+0000 7fb8e00a50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:53.017 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:53.016+0000 7fb8e00a50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:53.019 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:53.017+0000 7fb8e00a50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:53.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T09:16:53.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:16:53.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:16:53.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:16:53.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:16:53.374 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:53.374 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:16:53.374 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:16:53.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:53.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:16:53.534 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:53.532+0000 7fb8e00a50c0 -1 Falling back to public interface 2021-05-12T09:16:53.772 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/86471,v1:127.0.0.1:6811/86471] [v2:127.0.0.1:6812/86471,v1:127.0.0.1:6813/86471] exists,up 103e5b12-c0eb-48ea-93b8-2e63fc158391 2021-05-12T09:16:53.776 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:16:53.776 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:16:53.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:16:53.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2021-05-12T09:16:53.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/ec-error-rollforward 2021-05-12T09:16:53.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:16:53.778 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T09:16:53.778 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:16:53.778 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/ec-error-rollforward/2 2021-05-12T09:16:53.778 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T09:16:53.779 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:16:53.779 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:16:53.779 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:16:53.779 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2021-05-12T09:16:53.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2021-05-12T09:16:53.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:16:53.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:16:53.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T09:16:53.780 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:16:53.781 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:16:53.781 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:16:53.781 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:16:53.781 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:53.782 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:16:53.782 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:53.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:53.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:16:53.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T09:16:53.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T09:16:53.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:16:53.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:16:53.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:53.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:16:53.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:16:53.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/ec-error-rollforward/2 2021-05-12T09:16:53.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T09:16:53.785 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:16:53.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:53.786 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/ec-error-rollforward/2/whoami 2021-05-12T09:16:53.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T09:16:53.787 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:16:53.787 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:16:53.787 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:16:53.794 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:53.792+0000 7fbe618b60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:53.795 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:53.794+0000 7fbe618b60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:53.797 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:53.795+0000 7fbe618b60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:54.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T09:16:54.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:16:54.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:16:54.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:16:54.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:16:54.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:54.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:16:54.153 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:16:54.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:54.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:16:54.318 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:54.316+0000 7fbe618b60c0 -1 Falling back to public interface 2021-05-12T09:16:54.531 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/87856,v1:127.0.0.1:6819/87856] [v2:127.0.0.1:6820/87856,v1:127.0.0.1:6821/87856] exists,up 2242a79b-33a7-43a0-a01a-80c852755063 2021-05-12T09:16:54.534 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:16:54.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:16:54.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:16:54.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2021-05-12T09:16:54.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/ec-error-rollforward 2021-05-12T09:16:54.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:16:54.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=3 2021-05-12T09:16:54.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:16:54.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/ec-error-rollforward/3 2021-05-12T09:16:54.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-05-12T09:16:54.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:16:54.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:16:54.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:16:54.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2021-05-12T09:16:54.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2021-05-12T09:16:54.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:16:54.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:16:54.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-05-12T09:16:54.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:16:54.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:16:54.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:16:54.539 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:16:54.539 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:16:54.540 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:16:54.540 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:54.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' 2021-05-12T09:16:54.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:16:54.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-05-12T09:16:54.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-05-12T09:16:54.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:16:54.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:16:54.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:54.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:16:54.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:16:54.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/ec-error-rollforward/3 2021-05-12T09:16:54.542 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.3 2021-05-12T09:16:54.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.3 2021-05-12T09:16:54.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 3 --fsid=3a01f11c-42cc-455f-ae59-79615002e5a1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83966/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:16:54.544 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/ec-error-rollforward/3/whoami 2021-05-12T09:16:54.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 3 = 3 ']' 2021-05-12T09:16:54.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:16:54.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:16:54.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:16:54.553 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:54.551+0000 7f6ad8eff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:54.554 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:54.553+0000 7f6ad8eff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:54.555 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:54.554+0000 7f6ad8eff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:16:54.937 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 3 2021-05-12T09:16:54.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:16:54.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T09:16:54.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:16:54.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:16:54.939 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:16:54.939 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:16:54.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:16:54.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:16:54.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:16:55.072 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:55.070+0000 7f6ad8eff0c0 -1 Falling back to public interface 2021-05-12T09:16:55.314 INFO:tasks.workunit.client.0.smithi028.stdout:osd.3 up in weight 1 up_from 23 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/89244,v1:127.0.0.1:6827/89244] [v2:127.0.0.1:6828/89244,v1:127.0.0.1:6829/89244] exists,up 43f7f278-823e-4a0b-972d-59bbacc89ce8 2021-05-12T09:16:55.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:16:55.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:16:55.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:16:55.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2021-05-12T09:16:55.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:16:55.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:16:55.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:16:55.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:16:55.318 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T09:16:55.318 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:16:55.319 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:16:55.319 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:16:55.319 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:16:55.320 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:16:55.401 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:16:55.401 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:16:55.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:16:55.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:16:55.402 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:16:55.568 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:55.566+0000 7fd71cfcb0c0 -1 osd.0 29 log_to_monitors {default=true} 2021-05-12T09:16:55.787 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:16:55.787 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:16:55.788 INFO:tasks.workunit.client.0.smithi028.stderr:2 2021-05-12T09:16:55.788 INFO:tasks.workunit.client.0.smithi028.stderr:3' 2021-05-12T09:16:55.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:16:55.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:16:55.789 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:16:56.707 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:56.705+0000 7fb8e00a50c0 -1 osd.1 29 log_to_monitors {default=true} 2021-05-12T09:16:57.787 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:57.785+0000 7fbe618b60c0 -1 osd.2 28 log_to_monitors {default=true} 2021-05-12T09:16:58.592 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:16:58.590+0000 7f6ad8eff0c0 -1 osd.3 29 log_to_monitors {default=true} 2021-05-12T09:17:00.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=137438953476 2021-05-12T09:17:00.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 137438953476 2021-05-12T09:17:00.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-137438953476' 2021-05-12T09:17:00.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:17:00.927 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:17:01.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=150323855363 2021-05-12T09:17:01.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 150323855363 2021-05-12T09:17:01.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-137438953476 1-150323855363' 2021-05-12T09:17:01.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:17:01.064 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:17:01.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=167503724547 2021-05-12T09:17:01.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 167503724547 2021-05-12T09:17:01.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-137438953476 1-150323855363 2-167503724547' 2021-05-12T09:17:01.194 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:17:01.194 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T09:17:01.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=184683593731 2021-05-12T09:17:01.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 184683593731 2021-05-12T09:17:01.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-137438953476 1-150323855363 2-167503724547 3-184683593731' 2021-05-12T09:17:01.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:17:01.325 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-137438953476 2021-05-12T09:17:01.325 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:17:01.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:17:01.326 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-137438953476 2021-05-12T09:17:01.326 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:17:01.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=137438953476 2021-05-12T09:17:01.327 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 137438953476 2021-05-12T09:17:01.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 137438953476' 2021-05-12T09:17:01.328 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:17:01.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 137438953476 -lt 137438953476 2021-05-12T09:17:01.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:17:01.717 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-150323855363 2021-05-12T09:17:01.717 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:17:01.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:17:01.719 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-150323855363 2021-05-12T09:17:01.719 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:17:01.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=150323855363 2021-05-12T09:17:01.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 150323855363' 2021-05-12T09:17:01.720 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 150323855363 2021-05-12T09:17:01.721 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:17:02.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 150323855363 -lt 150323855363 2021-05-12T09:17:02.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:17:02.102 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-167503724547 2021-05-12T09:17:02.102 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:17:02.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:17:02.104 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-167503724547 2021-05-12T09:17:02.104 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:17:02.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=167503724547 2021-05-12T09:17:02.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 167503724547' 2021-05-12T09:17:02.105 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 167503724547 2021-05-12T09:17:02.106 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:17:02.482 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 167503724547 -lt 167503724547 2021-05-12T09:17:02.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:17:02.484 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-184683593731 2021-05-12T09:17:02.484 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:17:02.485 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T09:17:02.485 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-184683593731 2021-05-12T09:17:02.486 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:17:02.487 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=184683593731 2021-05-12T09:17:02.487 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 184683593731' 2021-05-12T09:17:02.487 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.3 seq 184683593731 2021-05-12T09:17:02.488 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T09:17:02.864 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 184683593731 -lt 184683593731 2021-05-12T09:17:02.865 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:17:02.865 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:17:02.866 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:17:03.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:17:03.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:17:03.381 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:17:03.382 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:17:03.382 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:17:03.382 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:17:03.382 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:17:03.383 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:17:03.748 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T09:17:03.749 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:17:03.750 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:17:03.750 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:17:04.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T09:17:04.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:17:04.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:17:04.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2021-05-12T09:17:04.246 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-05-12T09:17:04.246 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:17:04.246 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-05-12T09:17:04.247 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:17:04.247 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:17:04.247 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:17:04.248 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:17:04.248 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:17:04.355 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:17:04.356 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:17:04.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:17:04.357 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:17:04.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:17:04.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:17:04.360 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:17:04.360 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:17:04.360 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:17:04.361 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:17:04.361 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:17:04.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:17:04.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:17:04.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-05-12T09:17:04.368 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:17:04.368 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:17:04.368 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:17:04.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83966 2021-05-12T09:17:04.370 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:17:04.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:17:04.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T09:17:04.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/ec-error-rollforward 0 2021-05-12T09:17:04.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-05-12T09:17:04.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T09:17:04.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-05-12T09:17:04.372 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:17:04.372 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:17:04.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:17:04.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:17:04.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:17:04.374 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:17:04.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:17:04.374 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:17:04.375 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:17:04.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:17:04.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:17:04.375 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:17:04.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:17:04.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:17:04.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:17:04.377 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:17:04.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:17:04.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T09:17:04.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-05-12T09:17:04.379 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:17:04.379 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:17:04.379 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83966 2021-05-12T09:17:04.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83966 2021-05-12T09:17:04.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:17:04.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:17:04.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T09:17:04.382 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T09:17:04.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T09:17:04.453 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2021-05-12T09:17:04.454 DEBUG:teuthology.orchestra.run.smithi028:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2021-05-12T09:17:04.514 INFO:tasks.workunit.client.0.smithi028.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T09:17:04.522 INFO:tasks.workunit.client.0.smithi028.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T09:17:04.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:17:04.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:17:04.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T09:17:04.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T09:17:04.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T09:17:04.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T09:17:04.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T09:17:04.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T09:17:04.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-bench 2021-05-12T09:17:04.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2021-05-12T09:17:04.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2021-05-12T09:17:04.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2021-05-12T09:17:04.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2021-05-12T09:17:04.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2021-05-12T09:17:04.527 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2021-05-12T09:17:04.527 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=4ece8334-087e-4704-b94d-6849e73d87ba --auth-supported=none ' 2021-05-12T09:17:04.527 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2021-05-12T09:17:04.527 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2021-05-12T09:17:04.528 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2021-05-12T09:17:04.528 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T09:17:04.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2021-05-12T09:17:04.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2021-05-12T09:17:04.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2021-05-12T09:17:04.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2021-05-12T09:17:04.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2021-05-12T09:17:04.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-05-12T09:17:04.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:17:04.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-05-12T09:17:04.534 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:17:04.534 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:17:04.534 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:17:04.534 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:17:04.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:17:04.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:17:04.535 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:17:04.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:17:04.535 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:17:04.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:17:04.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:17:04.536 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:17:04.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:17:04.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:17:04.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:17:04.538 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:17:04.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:17:04.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:17:04.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-05-12T09:17:04.538 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:17:04.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:17:04.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92502 2021-05-12T09:17:04.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92502 2021-05-12T09:17:04.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:17:04.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:17:04.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2021-05-12T09:17:04.543 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:17:04.543 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:17:04.544 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92502 2021-05-12T09:17:04.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.92502 2021-05-12T09:17:04.544 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:17:04.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T09:17:04.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T09:17:04.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:17:04.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2021-05-12T09:17:04.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2021-05-12T09:17:04.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2021-05-12T09:17:04.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2021-05-12T09:17:04.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2021-05-12T09:17:04.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:17:04.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:17:04.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:17:04.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2021-05-12T09:17:04.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2021-05-12T09:17:04.881 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:17:04.881 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:17:04.882 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:17:04.882 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:17:04.882 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:17:04.883 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92502 2021-05-12T09:17:04.883 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92502/$cluster-$name.asok' 2021-05-12T09:17:04.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.92502/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:17:05.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:17:05.101 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:17:05.101 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:17:05.102 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:17:05.102 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:17:05.104 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:17:05.105 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:17:05.105 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:17:05.105 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:17:05.106 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:17:05.106 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:17:05.106 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92502 2021-05-12T09:17:05.106 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92502/ceph-mon.a.asok 2021-05-12T09:17:05.106 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:17:05.107 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.92502/ceph-mon.a.asok config get fsid 2021-05-12T09:17:05.201 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:17:05.201 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:17:05.202 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:17:05.202 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:17:05.202 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:17:05.202 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:17:05.202 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:17:05.203 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:17:05.203 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:17:05.203 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:17:05.203 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92502 2021-05-12T09:17:05.204 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92502/ceph-mon.a.asok 2021-05-12T09:17:05.204 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:17:05.204 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.92502/ceph-mon.a.asok config get mon_host 2021-05-12T09:17:05.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2021-05-12T09:17:05.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2021-05-12T09:17:05.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:17:05.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:17:05.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:17:05.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2021-05-12T09:17:05.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:17:06.267 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:17:06.268 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:17:06.268 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:17:06.269 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:17:06.269 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:17:06.270 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92502 2021-05-12T09:17:06.270 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92502/$cluster-$name.asok' 2021-05-12T09:17:06.270 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:17:06.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.92502/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:17:06.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2021-05-12T09:17:06.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2021-05-12T09:17:06.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:17:06.295 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:17:06.295 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:17:06.295 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2021-05-12T09:17:06.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4ece8334-087e-4704-b94d-6849e73d87ba --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2021-05-12T09:17:06.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:17:06.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:17:06.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:17:06.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2021-05-12T09:17:06.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2021-05-12T09:17:06.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:17:06.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:17:06.298 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2021-05-12T09:17:06.298 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:17:06.298 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:17:06.298 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:17:06.299 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:17:06.299 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:17:06.299 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92502 2021-05-12T09:17:06.299 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92502/$cluster-$name.asok' 2021-05-12T09:17:06.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.92502/$cluster-$name.asok' 2021-05-12T09:17:06.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:17:06.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:17:06.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:17:06.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2021-05-12T09:17:06.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2021-05-12T09:17:06.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:17:06.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:17:06.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:17:06.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:17:06.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:17:06.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bench/0 2021-05-12T09:17:06.302 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:17:06.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=aed5e565-9b98-4a1d-86f9-dabbb27331a7 2021-05-12T09:17:06.303 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 aed5e565-9b98-4a1d-86f9-dabbb27331a7' 2021-05-12T09:17:06.303 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 aed5e565-9b98-4a1d-86f9-dabbb27331a7 2021-05-12T09:17:06.304 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:17:06.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQASnZtgarIxEhAA6f9w/NU4HvqMnDZZRh2REg== 2021-05-12T09:17:06.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQASnZtgarIxEhAA6f9w/NU4HvqMnDZZRh2REg=="}' 2021-05-12T09:17:06.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new aed5e565-9b98-4a1d-86f9-dabbb27331a7 -i td/osd-bench/0/new.json 2021-05-12T09:17:06.624 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:17:06.636 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bench/0/new.json 2021-05-12T09:17:06.636 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=4ece8334-087e-4704-b94d-6849e73d87ba --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.92502/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQASnZtgarIxEhAA6f9w/NU4HvqMnDZZRh2REg== --osd-uuid aed5e565-9b98-4a1d-86f9-dabbb27331a7 2021-05-12T09:17:06.659 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:17:06.656+0000 7fbd20d4e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:17:06.660 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:17:06.658+0000 7fbd20d4e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:17:06.662 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:17:06.659+0000 7fbd20d4e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:17:06.663 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:17:06.660+0000 7fbd20d4e0c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-05-12T09:17:06.663 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:17:06.660+0000 7fbd20d4e0c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-05-12T09:17:06.663 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:17:06.660+0000 7fbd20d4e0c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-05-12T09:17:06.706 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:17:06.703+0000 7fbd20d4e0c0 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2021-05-12T09:17:11.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bench/0/keyring 2021-05-12T09:17:11.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:17:11.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:17:11.300 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:17:11.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:17:11.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:17:11.804 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:17:11.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=4ece8334-087e-4704-b94d-6849e73d87ba --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.92502/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:17:11.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:17:11.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:17:11.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:17:11.824 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:17:11.821+0000 7f1d994e90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:17:11.826 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:17:11.823+0000 7f1d994e90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:17:11.827 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:17:11.824+0000 7f1d994e90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:17:12.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:17:12.181 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:17:12.181 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:17:12.181 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:17:12.181 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:17:12.181 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:17:12.182 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:17:12.182 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:17:12.183 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:17:12.183 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:17:12.343 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:17:12.340+0000 7f1d994e90c0 -1 Falling back to public interface 2021-05-12T09:17:12.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:17:13.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:17:13.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:17:13.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:17:13.581 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:17:13.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:17:13.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:17:13.971 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:17:14.550 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:17:14.548+0000 7f1d994e90c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:17:14.972 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:17:14.972 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:17:14.972 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:17:14.973 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:17:14.973 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:17:14.973 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:17:15.384 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/93622,v1:127.0.0.1:6803/93622] [v2:127.0.0.1:6804/93622,v1:127.0.0.1:6805/93622] exists,up aed5e565-9b98-4a1d-86f9-dabbb27331a7 2021-05-12T09:17:15.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:17:15.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:17:15.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:17:15.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2021-05-12T09:17:15.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2021-05-12T09:17:15.395 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2021-05-12T09:17:15.395 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2021-05-12T09:17:15.396 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2021-05-12T09:17:15.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2021-05-12T09:17:15.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2021-05-12T09:17:15.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2021-05-12T09:17:15.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2021-05-12T09:17:15.416 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2021-05-12T09:17:15.416 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2021-05-12T09:17:15.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2021-05-12T09:17:15.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2021-05-12T09:17:15.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: local dir=td/osd-bench 2021-05-12T09:17:15.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: shift 2021-05-12T09:17:15.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: local expected=osd_bench_max_block_size 2021-05-12T09:17:15.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: shift 2021-05-12T09:17:15.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: expect_failure: local success 2021-05-12T09:17:15.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: expect_failure: ceph tell osd.0 bench 1024 67108865 2021-05-12T09:17:15.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: expect_failure: success=false 2021-05-12T09:17:15.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: false 2021-05-12T09:17:15.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2021-05-12T09:17:15.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: expect_failure: return 0 2021-05-12T09:17:15.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2021-05-12T09:17:15.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2021-05-12T09:17:15.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2021-05-12T09:17:15.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: local dir=td/osd-bench 2021-05-12T09:17:15.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: shift 2021-05-12T09:17:15.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: local expected=bench_small_size_max_iops 2021-05-12T09:17:15.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: shift 2021-05-12T09:17:15.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: expect_failure: local success 2021-05-12T09:17:15.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: expect_failure: ceph tell osd.0 bench 3072001 1024 2021-05-12T09:17:15.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: expect_failure: success=false 2021-05-12T09:17:15.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: false 2021-05-12T09:17:15.648 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2021-05-12T09:17:15.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: expect_failure: return 0 2021-05-12T09:17:15.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2021-05-12T09:17:15.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2021-05-12T09:17:15.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2021-05-12T09:17:15.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: local dir=td/osd-bench 2021-05-12T09:17:15.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: shift 2021-05-12T09:17:15.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: local expected=osd_bench_large_size_max_throughput 2021-05-12T09:17:15.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: shift 2021-05-12T09:17:15.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: expect_failure: local success 2021-05-12T09:17:15.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2021-05-12T09:17:15.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: expect_failure: success=false 2021-05-12T09:17:15.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: false 2021-05-12T09:17:15.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2021-05-12T09:17:15.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: expect_failure: return 0 2021-05-12T09:17:15.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2021-05-12T09:18:14.934 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:18:14.935 INFO:tasks.workunit.client.0.smithi028.stdout: "bytes_written": 1073741824, 2021-05-12T09:18:14.935 INFO:tasks.workunit.client.0.smithi028.stdout: "blocksize": 4194304, 2021-05-12T09:18:14.935 INFO:tasks.workunit.client.0.smithi028.stdout: "elapsed_sec": 30.743596047, 2021-05-12T09:18:14.936 INFO:tasks.workunit.client.0.smithi028.stdout: "bytes_per_sec": 34925706.880824603, 2021-05-12T09:18:14.936 INFO:tasks.workunit.client.0.smithi028.stdout: "iops": 8.3269374086438663 2021-05-12T09:18:14.936 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:18:14.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2021-05-12T09:18:15.730 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:18:15.731 INFO:tasks.workunit.client.0.smithi028.stdout: "bytes_written": 10, 2021-05-12T09:18:15.731 INFO:tasks.workunit.client.0.smithi028.stdout: "blocksize": 4444, 2021-05-12T09:18:15.731 INFO:tasks.workunit.client.0.smithi028.stdout: "elapsed_sec": 0.049919877000000001, 2021-05-12T09:18:15.731 INFO:tasks.workunit.client.0.smithi028.stdout: "bytes_per_sec": 200.3210063999156, 2021-05-12T09:18:15.731 INFO:tasks.workunit.client.0.smithi028.stdout: "iops": 0.045076734113392351 2021-05-12T09:18:15.732 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:18:15.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2021-05-12T09:18:16.256 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:18:16.256 INFO:tasks.workunit.client.0.smithi028.stdout: "bytes_written": 1, 2021-05-12T09:18:16.257 INFO:tasks.workunit.client.0.smithi028.stdout: "blocksize": 14456, 2021-05-12T09:18:16.257 INFO:tasks.workunit.client.0.smithi028.stdout: "elapsed_sec": 0.041150171999999999, 2021-05-12T09:18:16.257 INFO:tasks.workunit.client.0.smithi028.stdout: "bytes_per_sec": 24.301234998483118, 2021-05-12T09:18:16.257 INFO:tasks.workunit.client.0.smithi028.stdout: "iops": 0.0016810483535198615 2021-05-12T09:18:16.257 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:18:16.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2021-05-12T09:18:16.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-05-12T09:18:16.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:18:16.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-05-12T09:18:16.271 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:18:16.272 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:18:16.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:18:16.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:18:16.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:18:16.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:18:16.379 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:18:16.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:18:16.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:18:16.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:18:16.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:18:16.381 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:18:16.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:18:16.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:18:16.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:18:16.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:18:16.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:18:16.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:18:16.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-05-12T09:18:16.799 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:18:16.800 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:18:16.800 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92502 2021-05-12T09:18:16.800 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92502 2021-05-12T09:18:16.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:18:16.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:18:16.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T09:18:16.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-bench 0 2021-05-12T09:18:16.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-05-12T09:18:16.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T09:18:16.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-05-12T09:18:16.802 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:18:16.802 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:18:16.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:18:16.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:18:16.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:18:16.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:18:16.805 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:18:16.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:18:16.805 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:18:16.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:18:16.806 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:18:16.806 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:18:16.806 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:18:16.806 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:18:16.806 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:18:16.807 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:18:16.808 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:18:16.808 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T09:18:16.808 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-05-12T09:18:16.809 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:18:16.809 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:18:16.809 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92502 2021-05-12T09:18:16.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92502 2021-05-12T09:18:16.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:18:16.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:18:16.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T09:18:16.811 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T09:18:16.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T09:18:16.876 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2021-05-12T09:18:16.877 DEBUG:teuthology.orchestra.run.smithi028:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2021-05-12T09:18:16.937 INFO:tasks.workunit.client.0.smithi028.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T09:18:16.947 INFO:tasks.workunit.client.0.smithi028.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T09:18:16.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:18:16.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:18:16.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T09:18:16.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T09:18:16.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T09:18:16.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T09:18:16.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T09:18:16.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T09:18:16.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-bluefs-volume-ops 2021-05-12T09:18:16.950 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:18:16.950 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2021-05-12T09:18:16.950 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: export CEPH_MON=127.0.0.1:7146 2021-05-12T09:18:16.950 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: CEPH_MON=127.0.0.1:7146 2021-05-12T09:18:16.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: export CEPH_ARGS 2021-05-12T09:18:16.951 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: uuidgen 2021-05-12T09:18:16.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: CEPH_ARGS+='--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none ' 2021-05-12T09:18:16.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2021-05-12T09:18:16.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2021-05-12T09:18:16.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:16: run: CEPH_ARGS+='--bluestore_block_db_create=true ' 2021-05-12T09:18:16.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:17: run: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2021-05-12T09:18:16.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:18: run: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2021-05-12T09:18:16.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:19: run: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2021-05-12T09:18:16.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: run: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2021-05-12T09:18:16.953 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: set 2021-05-12T09:18:16.953 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T09:18:16.956 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: local funcs=TEST_bluestore 2021-05-12T09:18:16.956 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: run: for func in $funcs 2021-05-12T09:18:16.956 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: run: setup td/osd-bluefs-volume-ops 2021-05-12T09:18:16.957 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:18:16.957 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2021-05-12T09:18:16.957 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:18:16.957 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:18:16.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-05-12T09:18:16.958 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:18:16.958 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:18:16.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:18:16.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:18:16.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:18:16.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:18:16.959 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:18:16.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:18:16.960 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:18:16.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:18:16.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:18:16.960 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:18:16.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:18:16.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:18:16.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:18:16.961 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:18:16.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:18:16.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:18:16.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-05-12T09:18:16.965 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:18:16.965 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:18:16.965 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:18:16.965 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94395 2021-05-12T09:18:16.965 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:18:16.966 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:18:16.966 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2021-05-12T09:18:16.966 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:18:16.966 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:18:16.967 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:18:16.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.94395 2021-05-12T09:18:16.967 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:18:16.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T09:18:16.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T09:18:16.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:18:16.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2021-05-12T09:18:16.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:24: run: TEST_bluestore td/osd-bluefs-volume-ops 2021-05-12T09:18:16.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:18:16.968 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: ulimit -n 2021-05-12T09:18:16.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: local flimit=4096 2021-05-12T09:18:16.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: '[' 4096 -lt 1536 ']' 2021-05-12T09:18:16.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2021-05-12T09:18:16.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:18:16.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:18:16.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:18:16.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:18:16.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2021-05-12T09:18:16.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2021-05-12T09:18:17.315 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:18:17.315 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:18:17.316 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:18:17.316 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:18:17.316 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:18:17.316 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:18:17.316 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:18:17.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:18:17.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:18:17.610 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:18:17.610 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:18:17.610 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:18:17.610 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:18:17.611 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:18:17.611 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:18:17.611 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:18:17.611 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:18:17.612 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:18:17.612 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:18:17.612 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:18:17.612 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94395/ceph-mon.a.asok 2021-05-12T09:18:17.612 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:18:17.613 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.94395/ceph-mon.a.asok config get fsid 2021-05-12T09:18:17.714 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:18:17.715 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:18:17.715 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:18:17.715 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:18:17.715 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:18:17.716 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:18:17.716 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:18:17.716 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:18:17.716 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:18:17.717 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:18:17.717 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:18:17.717 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94395/ceph-mon.a.asok 2021-05-12T09:18:17.717 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:18:17.717 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.94395/ceph-mon.a.asok config get mon_host 2021-05-12T09:18:17.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2021-05-12T09:18:17.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:18:17.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:18:17.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:18:17.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:18:17.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2021-05-12T09:18:17.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:18:18.297 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:18:18.297 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:18:18.298 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:18:18.298 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:18:18.298 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:18:18.298 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:18:18.298 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:18:18.299 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:18:18.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:18:18.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2021-05-12T09:18:18.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:18:18.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:18:18.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:18:18.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:18:18.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-05-12T09:18:18.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:18:18.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:18:18.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:18:18.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:18:18.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-05-12T09:18:18.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-05-12T09:18:18.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:18:18.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:18:18.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:18:18.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:18:18.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:18:18.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:18:18.327 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:18:18.327 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:18:18.328 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:18:18.328 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:18:18.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:18:18.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:18:18.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:18:18.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:18:18.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:18:18.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:18:18.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:18:18.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:18:18.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:18:18.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:18:18.331 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:18:18.331 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-05-12T09:18:18.331 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:18:18.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=4dec11cd-c5bb-4f63-9dcf-2ee80e825c62 2021-05-12T09:18:18.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 4dec11cd-c5bb-4f63-9dcf-2ee80e825c62' 2021-05-12T09:18:18.333 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 4dec11cd-c5bb-4f63-9dcf-2ee80e825c62 2021-05-12T09:18:18.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:18:18.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBanZtg1/QCFBAA/wtCc9BcjYvRc8jce6YGXg== 2021-05-12T09:18:18.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBanZtg1/QCFBAA/wtCc9BcjYvRc8jce6YGXg=="}' 2021-05-12T09:18:18.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 4dec11cd-c5bb-4f63-9dcf-2ee80e825c62 -i td/osd-bluefs-volume-ops/0/new.json 2021-05-12T09:18:18.649 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:18:18.660 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2021-05-12T09:18:18.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBanZtg1/QCFBAA/wtCc9BcjYvRc8jce6YGXg== --osd-uuid 4dec11cd-c5bb-4f63-9dcf-2ee80e825c62 2021-05-12T09:18:18.685 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:18.681+0000 7fb5543f70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:18.686 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:18.683+0000 7fb5543f70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:18.687 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:18.684+0000 7fb5543f70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:18.688 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:18.685+0000 7fb5543f70c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-05-12T09:18:18.688 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:18.685+0000 7fb5543f70c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-05-12T09:18:18.688 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:18.685+0000 7fb5543f70c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-05-12T09:18:18.731 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:18.728+0000 7fb5543f70c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2021-05-12T09:18:27.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2021-05-12T09:18:27.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:18:27.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:18:27.060 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:18:27.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:18:27.570 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:18:27.571 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:18:27.571 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:18:27.572 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:18:27.572 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:18:27.572 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:18:27.590 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:27.589+0000 7fe4c0d410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:27.592 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:27.590+0000 7fe4c0d410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:27.593 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:27.592+0000 7fe4c0d410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:27.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:18:27.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:18:27.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:18:27.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:18:27.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:18:27.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:18:27.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:18:27.950 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:18:27.950 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:18:27.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:18:28.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:18:28.609 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:28.607+0000 7fe4c0d410c0 -1 Falling back to public interface 2021-05-12T09:18:29.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:18:29.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:18:29.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:18:29.341 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:18:29.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:18:29.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:18:29.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:18:30.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:18:30.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:18:30.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:18:30.729 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:18:30.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:18:30.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:18:31.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:18:32.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:18:32.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:18:32.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:18:32.117 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:18:32.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:18:32.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:18:32.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:18:33.066 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:33.065+0000 7fe4c0d410c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:18:33.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:18:33.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:18:33.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T09:18:33.503 INFO:tasks.workunit.client.0.smithi028.stdout:4 2021-05-12T09:18:33.504 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:18:33.505 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:18:33.911 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/95912,v1:127.0.0.1:6803/95912] [v2:127.0.0.1:6804/95912,v1:127.0.0.1:6805/95912] exists,up 4dec11cd-c5bb-4f63-9dcf-2ee80e825c62 2021-05-12T09:18:33.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:18:33.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:18:33.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:18:33.918 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-05-12T09:18:33.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=95912 2021-05-12T09:18:33.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2021-05-12T09:18:33.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:18:33.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:18:33.920 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:18:33.920 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:18:33.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-05-12T09:18:33.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:18:33.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:18:33.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:18:33.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:18:33.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-05-12T09:18:33.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-05-12T09:18:33.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:18:33.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:18:33.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:18:33.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:18:33.924 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:18:33.924 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:18:33.924 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:18:33.925 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:18:33.925 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:18:33.925 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:18:33.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:18:33.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:18:33.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:18:33.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:18:33.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:18:33.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:18:33.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:18:33.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:18:33.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:18:33.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:18:33.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:18:33.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-05-12T09:18:33.930 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:18:33.930 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=cb3779ec-96c0-4e85-9aa4-c83b3e395169 2021-05-12T09:18:33.930 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 cb3779ec-96c0-4e85-9aa4-c83b3e395169' 2021-05-12T09:18:33.939 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 cb3779ec-96c0-4e85-9aa4-c83b3e395169 2021-05-12T09:18:33.940 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:18:33.941 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBpnZtgq/pGNxAARPbpX+RukR/ISf4fTm1iqQ== 2021-05-12T09:18:33.941 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBpnZtgq/pGNxAARPbpX+RukR/ISf4fTm1iqQ=="}' 2021-05-12T09:18:33.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new cb3779ec-96c0-4e85-9aa4-c83b3e395169 -i td/osd-bluefs-volume-ops/1/new.json 2021-05-12T09:18:34.438 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:18:34.449 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2021-05-12T09:18:34.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBpnZtgq/pGNxAARPbpX+RukR/ISf4fTm1iqQ== --osd-uuid cb3779ec-96c0-4e85-9aa4-c83b3e395169 2021-05-12T09:18:34.472 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:34.471+0000 7f617c24b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:34.474 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:34.472+0000 7f617c24b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:34.475 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:34.473+0000 7f617c24b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:34.476 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:34.474+0000 7f617c24b0c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-05-12T09:18:34.476 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:34.474+0000 7f617c24b0c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-05-12T09:18:34.476 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:34.474+0000 7f617c24b0c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-05-12T09:18:34.512 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:34.511+0000 7f617c24b0c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2021-05-12T09:18:43.132 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2021-05-12T09:18:43.132 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:18:43.133 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:18:43.134 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:18:43.134 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:18:43.645 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:18:43.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:18:43.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:18:43.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:18:43.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:18:43.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:18:43.666 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:43.664+0000 7f360c5e20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:43.667 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:43.666+0000 7f360c5e20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:43.669 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:43.668+0000 7f360c5e20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:44.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:18:44.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:18:44.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:18:44.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:18:44.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:18:44.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:18:44.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:18:44.033 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:18:44.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:18:44.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:18:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:18:44.690 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:44.689+0000 7f360c5e20c0 -1 Falling back to public interface 2021-05-12T09:18:45.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:18:45.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:18:45.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:18:45.422 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:18:45.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:18:45.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:18:45.811 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:18:46.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:18:46.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:18:46.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:18:46.814 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:18:46.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:18:46.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:18:47.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:18:48.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:18:48.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:18:48.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:18:48.227 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:18:48.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:18:48.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:18:48.613 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:18:49.134 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:49.131+0000 7f360c5e20c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:18:49.615 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:18:49.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:18:49.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T09:18:49.617 INFO:tasks.workunit.client.0.smithi028.stdout:4 2021-05-12T09:18:49.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:18:49.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:18:50.025 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/98132,v1:127.0.0.1:6811/98132] [v2:127.0.0.1:6812/98132,v1:127.0.0.1:6813/98132] exists,up cb3779ec-96c0-4e85-9aa4-c83b3e395169 2021-05-12T09:18:50.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:18:50.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:18:50.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:18:50.027 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-05-12T09:18:50.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=98132 2021-05-12T09:18:50.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2021-05-12T09:18:50.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:18:50.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:18:50.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T09:18:50.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:18:50.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-05-12T09:18:50.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:18:50.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:18:50.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:18:50.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:18:50.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-05-12T09:18:50.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-05-12T09:18:50.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:18:50.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:18:50.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:18:50.036 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:18:50.036 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:18:50.037 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:18:50.037 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:18:50.037 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:18:50.037 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:18:50.038 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:18:50.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:18:50.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:18:50.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:18:50.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:18:50.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:18:50.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:18:50.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:18:50.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:18:50.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:18:50.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:18:50.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:18:50.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-05-12T09:18:50.041 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:18:50.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=5bffcfc9-40a9-4ed4-900b-d2358b9b5f65 2021-05-12T09:18:50.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 5bffcfc9-40a9-4ed4-900b-d2358b9b5f65' 2021-05-12T09:18:50.042 INFO:tasks.workunit.client.0.smithi028.stdout:add osd2 5bffcfc9-40a9-4ed4-900b-d2358b9b5f65 2021-05-12T09:18:50.042 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:18:50.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQB6nZtgsgiiAhAAaSUya4tgWEUZPjNZua5UhA== 2021-05-12T09:18:50.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQB6nZtgsgiiAhAAaSUya4tgWEUZPjNZua5UhA=="}' 2021-05-12T09:18:50.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 5bffcfc9-40a9-4ed4-900b-d2358b9b5f65 -i td/osd-bluefs-volume-ops/2/new.json 2021-05-12T09:18:50.523 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:18:50.534 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2021-05-12T09:18:50.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB6nZtgsgiiAhAAaSUya4tgWEUZPjNZua5UhA== --osd-uuid 5bffcfc9-40a9-4ed4-900b-d2358b9b5f65 2021-05-12T09:18:50.557 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:50.554+0000 7fa9048930c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:50.559 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:50.556+0000 7fa9048930c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:50.559 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:50.557+0000 7fa9048930c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:50.560 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:50.557+0000 7fa9048930c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-05-12T09:18:50.560 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:50.557+0000 7fa9048930c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-05-12T09:18:50.561 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:50.557+0000 7fa9048930c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-05-12T09:18:50.603 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:50.599+0000 7fa9048930c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2021-05-12T09:18:59.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2021-05-12T09:18:59.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:18:59.319 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T09:18:59.320 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd2 key to auth repository 2021-05-12T09:18:59.320 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:18:59.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T09:18:59.831 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:18:59.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:18:59.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:18:59.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:18:59.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:18:59.851 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:59.848+0000 7f01be6280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:59.853 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:59.850+0000 7f01be6280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:18:59.854 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:18:59.852+0000 7f01be6280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:19:00.211 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T09:19:00.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:19:00.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:19:00.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:19:00.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:19:00.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:19:00.213 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:19:00.213 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:19:00.214 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:19:00.214 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:19:00.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:19:00.870 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:19:00.868+0000 7f01be6280c0 -1 Falling back to public interface 2021-05-12T09:19:01.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:19:01.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:19:01.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:19:01.606 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:19:01.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:19:01.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:19:01.995 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:19:02.996 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:19:02.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:19:02.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:19:02.998 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:19:02.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:19:02.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:19:03.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:19:04.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:19:04.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:19:04.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:19:04.388 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:19:04.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:19:04.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:19:04.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:19:05.355 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:19:05.353+0000 7f01be6280c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T09:19:05.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:19:05.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:19:05.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T09:19:05.775 INFO:tasks.workunit.client.0.smithi028.stdout:4 2021-05-12T09:19:05.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:19:05.776 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:19:06.186 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/100352,v1:127.0.0.1:6819/100352] [v2:127.0.0.1:6820/100352,v1:127.0.0.1:6821/100352] exists,up 5bffcfc9-40a9-4ed4-900b-d2358b9b5f65 2021-05-12T09:19:06.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:19:06.192 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:19:06.192 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:19:06.193 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-05-12T09:19:06.197 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=100352 2021-05-12T09:19:06.197 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2021-05-12T09:19:06.197 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:19:06.197 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:19:06.198 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-05-12T09:19:06.198 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:19:06.198 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-05-12T09:19:06.198 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:19:06.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:19:06.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:19:06.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:19:06.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-05-12T09:19:06.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-05-12T09:19:06.200 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:19:06.200 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:19:06.200 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:19:06.200 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:19:06.200 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:19:06.201 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:19:06.201 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:19:06.201 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:19:06.201 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:19:06.202 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:19:06.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:19:06.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:19:06.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:19:06.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:19:06.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:19:06.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:19:06.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:19:06.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:19:06.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:19:06.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:19:06.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:19:06.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-05-12T09:19:06.204 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:19:06.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=da2b28e2-81ee-4600-aaab-71063cad2c8c 2021-05-12T09:19:06.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 da2b28e2-81ee-4600-aaab-71063cad2c8c' 2021-05-12T09:19:06.205 INFO:tasks.workunit.client.0.smithi028.stdout:add osd3 da2b28e2-81ee-4600-aaab-71063cad2c8c 2021-05-12T09:19:06.206 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:19:06.206 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCKnZtgySIrDBAANhd0Ys8c5VFpLWBykUayXg== 2021-05-12T09:19:06.206 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCKnZtgySIrDBAANhd0Ys8c5VFpLWBykUayXg=="}' 2021-05-12T09:19:06.207 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new da2b28e2-81ee-4600-aaab-71063cad2c8c -i td/osd-bluefs-volume-ops/3/new.json 2021-05-12T09:19:06.790 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:19:06.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2021-05-12T09:19:06.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCKnZtgySIrDBAANhd0Ys8c5VFpLWBykUayXg== --osd-uuid da2b28e2-81ee-4600-aaab-71063cad2c8c 2021-05-12T09:19:06.824 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:19:06.823+0000 7f590e14c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:19:06.826 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:19:06.825+0000 7f590e14c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:19:06.828 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:19:06.826+0000 7f590e14c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:19:06.829 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:19:06.827+0000 7f590e14c0c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-05-12T09:19:06.829 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:19:06.827+0000 7f590e14c0c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-05-12T09:19:06.829 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:19:06.827+0000 7f590e14c0c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-05-12T09:19:06.868 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:19:06.866+0000 7f590e14c0c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2021-05-12T09:19:18.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2021-05-12T09:19:18.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:19:18.141 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2021-05-12T09:19:18.141 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd3 key to auth repository 2021-05-12T09:19:18.142 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:19:18.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2021-05-12T09:19:18.659 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.3 2021-05-12T09:19:18.660 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:19:18.660 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:19:18.660 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:19:18.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:19:18.679 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:19:18.677+0000 7f3c298e00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:19:18.681 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:19:18.679+0000 7f3c298e00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:19:18.682 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:19:18.681+0000 7f3c298e00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:19:19.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2021-05-12T09:19:19.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:19:19.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T09:19:19.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:19:19.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:19:19.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:19:19.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:19:19.040 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:19:19.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:19:19.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:19:19.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:19:19.698 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:19:19.697+0000 7f3c298e00c0 -1 Falling back to public interface 2021-05-12T09:19:20.433 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:19:20.433 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:19:20.434 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:19:20.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:19:20.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:19:20.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:19:20.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:19:21.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:19:21.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:19:21.823 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:19:21.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:19:21.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:19:21.825 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:19:22.211 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:19:23.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:19:23.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:19:23.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:19:23.213 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:19:23.213 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:19:23.214 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:19:23.613 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:19:24.214 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:19:24.211+0000 7f3c298e00c0 -1 osd.3 0 log_to_monitors {default=true} 2021-05-12T09:19:24.615 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:19:24.615 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:19:24.615 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T09:19:24.616 INFO:tasks.workunit.client.0.smithi028.stdout:4 2021-05-12T09:19:24.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:19:24.617 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:19:25.052 INFO:tasks.workunit.client.0.smithi028.stdout:osd.3 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/102583,v1:127.0.0.1:6827/102583] [v2:127.0.0.1:6828/102583,v1:127.0.0.1:6829/102583] exists,up da2b28e2-81ee-4600-aaab-71063cad2c8c 2021-05-12T09:19:25.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:19:25.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:19:25.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:19:25.054 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-05-12T09:19:25.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=102583 2021-05-12T09:19:25.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2021-05-12T09:19:30.055 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2021-05-12T09:19:30.055 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2021-05-12T09:19:30.599 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'foo' already exists 2021-05-12T09:19:30.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:19:31.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-05-12T09:19:31.687 INFO:tasks.workunit.client.0.smithi028.stdout:hints = 1 2021-05-12T09:19:31.688 INFO:tasks.workunit.client.0.smithi028.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-05-12T09:19:31.688 INFO:tasks.workunit.client.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_103593 2021-05-12T09:19:31.688 INFO:tasks.workunit.client.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T09:19:31.689 INFO:tasks.workunit.client.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2021-05-12T09:19:32.688 INFO:tasks.workunit.client.0.smithi028.stdout: 1 16 42 26 0.10155 0.101562 0.27826 0.480978 2021-05-12T09:19:33.688 INFO:tasks.workunit.client.0.smithi028.stdout: 2 16 97 81 0.15818 0.214844 0.284897 0.362457 2021-05-12T09:19:34.688 INFO:tasks.workunit.client.0.smithi028.stdout: 3 16 161 145 0.188774 0.25 0.284894 0.318143 2021-05-12T09:19:35.689 INFO:tasks.workunit.client.0.smithi028.stdout: 4 16 223 207 0.202118 0.242188 0.29207 0.301612 2021-05-12T09:19:36.690 INFO:tasks.workunit.client.0.smithi028.stdout: 5 16 279 263 0.20544 0.21875 0.310141 0.296576 2021-05-12T09:19:37.688 INFO:tasks.workunit.client.0.smithi028.stdout: 6 16 337 321 0.208955 0.226562 0.276944 0.293646 2021-05-12T09:19:38.689 INFO:tasks.workunit.client.0.smithi028.stdout: 7 16 401 385 0.214814 0.25 0.251864 0.28674 2021-05-12T09:19:39.689 INFO:tasks.workunit.client.0.smithi028.stdout: 8 16 465 449 0.219209 0.25 0.242629 0.282386 2021-05-12T09:19:40.689 INFO:tasks.workunit.client.0.smithi028.stdout: 9 16 529 513 0.222627 0.25 0.218514 0.277144 2021-05-12T09:19:41.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 16 583 567 0.221455 0.210938 0.268767 0.277573 2021-05-12T09:19:42.690 INFO:tasks.workunit.client.0.smithi028.stdout: 11 16 655 639 0.226889 0.28125 0.244206 0.27365 2021-05-12T09:19:43.689 INFO:tasks.workunit.client.0.smithi028.stdout: 12 16 711 695 0.226208 0.21875 0.298112 0.272331 2021-05-12T09:19:44.689 INFO:tasks.workunit.client.0.smithi028.stdout: 13 16 775 759 0.228036 0.25 0.26802 0.271172 2021-05-12T09:19:45.689 INFO:tasks.workunit.client.0.smithi028.stdout: 14 16 836 820 0.228765 0.238281 0.241336 0.270498 2021-05-12T09:19:46.691 INFO:tasks.workunit.client.0.smithi028.stdout: 15 16 900 884 0.230179 0.25 0.276989 0.269154 2021-05-12T09:19:47.690 INFO:tasks.workunit.client.0.smithi028.stdout: 16 16 964 948 0.231416 0.25 0.283414 0.267696 2021-05-12T09:19:48.690 INFO:tasks.workunit.client.0.smithi028.stdout: 17 16 1028 1012 0.232506 0.25 0.23345 0.266688 2021-05-12T09:19:49.690 INFO:tasks.workunit.client.0.smithi028.stdout: 18 16 1089 1073 0.232825 0.238281 0.27309 0.266521 2021-05-12T09:19:50.691 INFO:tasks.workunit.client.0.smithi028.stdout: 19 16 1153 1137 0.233727 0.25 0.24355 0.265687 2021-05-12T09:19:51.690 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:19:51.689434+0000 min lat: 0.209119 max lat: 0.760676 avg lat: 0.267808 2021-05-12T09:19:51.691 INFO:tasks.workunit.client.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T09:19:51.691 INFO:tasks.workunit.client.0.smithi028.stdout: 20 16 1201 1185 0.231414 0.1875 0.276383 0.267808 2021-05-12T09:19:52.691 INFO:tasks.workunit.client.0.smithi028.stdout: 21 16 1265 1249 0.232297 0.25 0.258637 0.267734 2021-05-12T09:19:53.691 INFO:tasks.workunit.client.0.smithi028.stdout: 22 16 1319 1303 0.231325 0.210938 0.276298 0.26774 2021-05-12T09:19:54.691 INFO:tasks.workunit.client.0.smithi028.stdout: 23 16 1383 1367 0.232136 0.25 0.262975 0.267516 2021-05-12T09:19:55.691 INFO:tasks.workunit.client.0.smithi028.stdout: 24 16 1441 1425 0.231902 0.226562 0.251616 0.268059 2021-05-12T09:19:56.691 INFO:tasks.workunit.client.0.smithi028.stdout: 25 16 1495 1479 0.231062 0.210938 0.31028 0.268816 2021-05-12T09:19:57.691 INFO:tasks.workunit.client.0.smithi028.stdout: 26 16 1553 1537 0.230888 0.226562 0.300544 0.269215 2021-05-12T09:19:58.692 INFO:tasks.workunit.client.0.smithi028.stdout: 27 16 1615 1599 0.231305 0.242188 0.262545 0.269336 2021-05-12T09:19:59.691 INFO:tasks.workunit.client.0.smithi028.stdout: 28 16 1671 1655 0.230856 0.21875 0.249772 0.269289 2021-05-12T09:20:00.692 INFO:tasks.workunit.client.0.smithi028.stdout: 29 16 1735 1719 0.231515 0.25 0.250207 0.268867 2021-05-12T09:20:01.692 INFO:tasks.workunit.client.0.smithi028.stdout: 30 16 1793 1777 0.231349 0.226562 0.285184 0.26868 2021-05-12T09:20:02.692 INFO:tasks.workunit.client.0.smithi028.stdout:Total time run: 30.1653 2021-05-12T09:20:02.692 INFO:tasks.workunit.client.0.smithi028.stdout:Total writes made: 1793 2021-05-12T09:20:02.693 INFO:tasks.workunit.client.0.smithi028.stdout:Write size: 4096 2021-05-12T09:20:02.693 INFO:tasks.workunit.client.0.smithi028.stdout:Object size: 4096 2021-05-12T09:20:02.693 INFO:tasks.workunit.client.0.smithi028.stdout:Bandwidth (MB/sec): 0.232184 2021-05-12T09:20:02.693 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev Bandwidth: 0.030991 2021-05-12T09:20:02.693 INFO:tasks.workunit.client.0.smithi028.stdout:Max bandwidth (MB/sec): 0.28125 2021-05-12T09:20:02.694 INFO:tasks.workunit.client.0.smithi028.stdout:Min bandwidth (MB/sec): 0.101562 2021-05-12T09:20:02.695 INFO:tasks.workunit.client.0.smithi028.stdout:Average IOPS: 59 2021-05-12T09:20:02.696 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev IOPS: 7.93371 2021-05-12T09:20:02.696 INFO:tasks.workunit.client.0.smithi028.stdout:Max IOPS: 72 2021-05-12T09:20:02.696 INFO:tasks.workunit.client.0.smithi028.stdout:Min IOPS: 26 2021-05-12T09:20:02.696 INFO:tasks.workunit.client.0.smithi028.stdout:Average Latency(s): 0.268656 2021-05-12T09:20:02.696 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev Latency(s): 0.04696 2021-05-12T09:20:02.697 INFO:tasks.workunit.client.0.smithi028.stdout:Max latency(s): 0.760676 2021-05-12T09:20:02.697 INFO:tasks.workunit.client.0.smithi028.stdout:Min latency(s): 0.209119 2021-05-12T09:20:02.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2021-05-12T09:20:02.798 INFO:tasks.workunit.client.0.smithi028.stdout:after bench 2021-05-12T09:20:02.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 95912 2021-05-12T09:20:02.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2021-05-12T09:20:03.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 95912 2021-05-12T09:20:03.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (95912) - No such process 2021-05-12T09:20:03.800 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2021-05-12T09:20:04.203 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already down. 2021-05-12T09:20:04.213 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 98132 2021-05-12T09:20:04.213 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2021-05-12T09:20:05.214 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 98132 2021-05-12T09:20:05.215 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (98132) - No such process 2021-05-12T09:20:05.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2021-05-12T09:20:05.925 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already down. 2021-05-12T09:20:05.937 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 100352 2021-05-12T09:20:05.937 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2021-05-12T09:20:06.939 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 100352 2021-05-12T09:20:06.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (100352) - No such process 2021-05-12T09:20:06.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2021-05-12T09:20:07.661 INFO:tasks.workunit.client.0.smithi028.stderr:osd.2 is already down. 2021-05-12T09:20:07.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 102583 2021-05-12T09:20:07.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2021-05-12T09:20:08.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 102583 2021-05-12T09:20:08.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (102583) - No such process 2021-05-12T09:20:08.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2021-05-12T09:20:09.216 INFO:tasks.workunit.client.0.smithi028.stderr:osd.3 is already down. 2021-05-12T09:20:09.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T09:20:11.007 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:20:11.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T09:20:12.796 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:20:12.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T09:20:14.584 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:20:14.587 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T09:20:16.374 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:20:16.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2021-05-12T09:20:16.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2021-05-12T09:20:16.401 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:20:17.191 INFO:tasks.workunit.client.0.smithi028.stdout:0 : device size 0x1ffff000 : using 0xe00000(14 MiB) 2021-05-12T09:20:17.192 INFO:tasks.workunit.client.0.smithi028.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-05-12T09:20:17.192 INFO:tasks.workunit.client.0.smithi028.stdout:2 : device size 0x100000000 : using 0x8065a000(2.0 GiB) 2021-05-12T09:20:17.192 INFO:tasks.workunit.client.0.smithi028.stdout:Expanding DB/WAL... 2021-05-12T09:20:17.192 INFO:tasks.workunit.client.0.smithi028.stdout:0 : expanding to 0x536870912 2021-05-12T09:20:17.238 INFO:tasks.workunit.client.0.smithi028.stdout:0 : size label updated to 536870912 2021-05-12T09:20:17.238 INFO:tasks.workunit.client.0.smithi028.stdout:1 : expanding to 0x1073741824 2021-05-12T09:20:17.280 INFO:tasks.workunit.client.0.smithi028.stdout:1 : size label updated to 1073741824 2021-05-12T09:20:17.280 INFO:tasks.workunit.client.0.smithi028.stdout:2 : expanding from 0x80000000 to 0x100000000 2021-05-12T09:20:17.548 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:20:17.546+0000 7f010c3b7400 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/0: (21) Is a directory 2021-05-12T09:20:17.549 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:20:17.546+0000 7f010c3b7400 -1 bluestore(td/osd-bluefs-volume-ops/0) unable to read label for td/osd-bluefs-volume-ops/0: (21) Is a directory 2021-05-12T09:20:22.549 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2021-05-12T09:20:22.550 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2021-05-12T09:20:22.566 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:20:23.355 INFO:tasks.workunit.client.0.smithi028.stdout:0 : device size 0x1ffff000 : using 0xe00000(14 MiB) 2021-05-12T09:20:23.355 INFO:tasks.workunit.client.0.smithi028.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-05-12T09:20:23.355 INFO:tasks.workunit.client.0.smithi028.stdout:2 : device size 0x101000000 : using 0x8165c000(2.0 GiB) 2021-05-12T09:20:23.356 INFO:tasks.workunit.client.0.smithi028.stdout:Expanding DB/WAL... 2021-05-12T09:20:23.356 INFO:tasks.workunit.client.0.smithi028.stdout:0 : expanding to 0x536870912 2021-05-12T09:20:23.369 INFO:tasks.workunit.client.0.smithi028.stdout:0 : size label updated to 536870912 2021-05-12T09:20:23.369 INFO:tasks.workunit.client.0.smithi028.stdout:1 : expanding to 0x1073741824 2021-05-12T09:20:23.386 INFO:tasks.workunit.client.0.smithi028.stdout:1 : size label updated to 1073741824 2021-05-12T09:20:23.386 INFO:tasks.workunit.client.0.smithi028.stdout:2 : expanding from 0x80000000 to 0x101000000 2021-05-12T09:20:23.662 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:20:23.660+0000 7fc680c03400 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/1: (21) Is a directory 2021-05-12T09:20:23.662 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:20:23.660+0000 7fc680c03400 -1 bluestore(td/osd-bluefs-volume-ops/1) unable to read label for td/osd-bluefs-volume-ops/1: (21) Is a directory 2021-05-12T09:20:28.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2021-05-12T09:20:28.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2021-05-12T09:20:28.903 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:20:29.691 INFO:tasks.workunit.client.0.smithi028.stdout:0 : device size 0x1ffff000 : using 0xc00000(12 MiB) 2021-05-12T09:20:29.691 INFO:tasks.workunit.client.0.smithi028.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-05-12T09:20:29.691 INFO:tasks.workunit.client.0.smithi028.stdout:2 : device size 0x100020000 : using 0x8055c000(2.0 GiB) 2021-05-12T09:20:29.692 INFO:tasks.workunit.client.0.smithi028.stdout:Expanding DB/WAL... 2021-05-12T09:20:29.692 INFO:tasks.workunit.client.0.smithi028.stdout:0 : expanding to 0x536870912 2021-05-12T09:20:29.716 INFO:tasks.workunit.client.0.smithi028.stdout:0 : size label updated to 536870912 2021-05-12T09:20:29.717 INFO:tasks.workunit.client.0.smithi028.stdout:1 : expanding to 0x1073741824 2021-05-12T09:20:29.742 INFO:tasks.workunit.client.0.smithi028.stdout:1 : size label updated to 1073741824 2021-05-12T09:20:29.742 INFO:tasks.workunit.client.0.smithi028.stdout:2 : expanding from 0x80000000 to 0x100020000 2021-05-12T09:20:29.926 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:20:29.924+0000 7fdd044aa400 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/2: (21) Is a directory 2021-05-12T09:20:29.926 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:20:29.924+0000 7fdd044aa400 -1 bluestore(td/osd-bluefs-volume-ops/2) unable to read label for td/osd-bluefs-volume-ops/2: (21) Is a directory 2021-05-12T09:20:34.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2021-05-12T09:20:34.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2021-05-12T09:20:34.900 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:20:35.687 INFO:tasks.workunit.client.0.smithi028.stdout:0 : device size 0x1ffff000 : using 0xc00000(12 MiB) 2021-05-12T09:20:35.688 INFO:tasks.workunit.client.0.smithi028.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-05-12T09:20:35.688 INFO:tasks.workunit.client.0.smithi028.stdout:2 : device size 0xfff00000 : using 0x8049c000(2.0 GiB) 2021-05-12T09:20:35.688 INFO:tasks.workunit.client.0.smithi028.stdout:Expanding DB/WAL... 2021-05-12T09:20:35.689 INFO:tasks.workunit.client.0.smithi028.stdout:0 : expanding to 0x536870912 2021-05-12T09:20:35.722 INFO:tasks.workunit.client.0.smithi028.stdout:0 : size label updated to 536870912 2021-05-12T09:20:35.722 INFO:tasks.workunit.client.0.smithi028.stdout:1 : expanding to 0x1073741824 2021-05-12T09:20:35.747 INFO:tasks.workunit.client.0.smithi028.stdout:1 : size label updated to 1073741824 2021-05-12T09:20:35.747 INFO:tasks.workunit.client.0.smithi028.stdout:2 : expanding from 0x80000000 to 0xfff00000 2021-05-12T09:20:35.923 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:20:35.921+0000 7f3c3cbb4400 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/3: (21) Is a directory 2021-05-12T09:20:35.923 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:20:35.921+0000 7f3c3cbb4400 -1 bluestore(td/osd-bluefs-volume-ops/3) unable to read label for td/osd-bluefs-volume-ops/3: (21) Is a directory 2021-05-12T09:20:40.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T09:20:42.653 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:20:42.656 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T09:20:44.441 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:20:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T09:20:46.230 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:20:46.233 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T09:20:48.021 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:20:48.023 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2021-05-12T09:20:48.035 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:20:49.807 INFO:tasks.workunit.client.0.smithi028.stdout:0 : device size 0x1ffff000 : using 0x500000(5 MiB) 2021-05-12T09:20:49.808 INFO:tasks.workunit.client.0.smithi028.stdout:1 : device size 0x3fffe000 : using 0x1000000(16 MiB) 2021-05-12T09:20:49.808 INFO:tasks.workunit.client.0.smithi028.stdout:2 : device size 0x100000000 : using 0x65a000(6.4 MiB) 2021-05-12T09:20:49.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2021-05-12T09:20:49.824 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:20:51.343 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2021-05-12T09:20:51.346 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T09:20:52.630 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:20:52.633 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2021-05-12T09:20:52.648 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:20:54.421 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-05-12T09:20:54.424 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T09:20:55.709 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:20:55.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2021-05-12T09:20:55.726 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:20:57.245 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2021-05-12T09:20:57.246 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2021-05-12T09:20:57.248 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T09:20:58.035 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:20:58.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2021-05-12T09:20:58.053 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:20:58.058 INFO:tasks.workunit.client.0.smithi028.stderr:Migrate to WAL device isn't supported. 2021-05-12T09:20:58.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2021-05-12T09:20:58.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T09:20:59.845 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:20:59.847 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2021-05-12T09:20:59.862 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:21:01.633 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2021-05-12T09:21:01.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T09:21:02.920 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:21:02.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2021-05-12T09:21:02.936 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:21:04.209 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T09:21:05.495 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:21:05.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-05-12T09:21:05.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:21:05.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:21:05.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T09:21:05.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:21:05.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-05-12T09:21:05.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:21:05.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:21:05.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:21:05.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:21:05.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-05-12T09:21:05.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-05-12T09:21:05.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:21:05.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:21:05.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:21:05.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:21:05.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:21:05.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:21:05.504 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:21:05.504 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:21:05.504 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:21:05.505 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:21:05.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:21:05.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:21:05.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:21:05.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:21:05.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:21:05.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:21:05.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:21:05.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:21:05.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:21:05.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-05-12T09:21:05.511 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:21:05.513 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T09:21:05.513 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:21:05.513 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-05-12T09:21:05.513 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T09:21:05.514 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:21:05.514 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:21:05.514 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:21:05.525 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:05.523+0000 7f58c7da70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:21:05.527 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:05.525+0000 7f58c7da70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:21:05.528 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:05.527+0000 7f58c7da70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:21:05.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T09:21:05.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:21:05.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:21:05.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:21:05.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:21:05.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:05.891 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:21:05.891 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:21:05.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:05.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:21:06.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:21:06.298 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:06.296+0000 7f58c7da70c0 -1 Falling back to public interface 2021-05-12T09:21:07.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:21:07.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:07.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:21:07.272 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:21:07.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:07.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:21:07.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:21:08.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:21:08.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:08.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:21:08.675 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:21:08.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:08.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:21:09.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:21:10.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:21:10.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:10.060 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:21:10.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:21:10.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:10.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:21:10.221 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:10.218+0000 7f58c7da70c0 -1 osd.0 33 log_to_monitors {default=true} 2021-05-12T09:21:10.487 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:21:11.488 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:21:11.488 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:11.488 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T09:21:11.489 INFO:tasks.workunit.client.0.smithi028.stdout:4 2021-05-12T09:21:11.489 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:11.489 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:21:11.874 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 43 up_thru 43 down_at 34 last_clean_interval [5,33) [v2:127.0.0.1:6802/115881,v1:127.0.0.1:6803/115881] [v2:127.0.0.1:6804/115881,v1:127.0.0.1:6805/115881] exists,up 4dec11cd-c5bb-4f63-9dcf-2ee80e825c62 2021-05-12T09:21:11.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:21:11.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:21:11.876 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:21:11.876 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-05-12T09:21:11.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=115881 2021-05-12T09:21:11.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-05-12T09:21:11.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:21:11.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:21:11.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T09:21:11.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:21:11.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-05-12T09:21:11.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:21:11.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:21:11.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:21:11.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:21:11.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-05-12T09:21:11.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-05-12T09:21:11.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:21:11.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:21:11.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:21:11.883 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:21:11.884 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:21:11.884 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:21:11.884 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:21:11.884 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:21:11.885 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:21:11.885 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:21:11.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:21:11.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:21:11.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:21:11.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:21:11.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:21:11.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:21:11.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:21:11.887 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:21:11.887 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:21:11.887 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-05-12T09:21:11.887 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:21:11.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T09:21:11.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:21:11.889 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-05-12T09:21:11.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T09:21:11.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:21:11.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:21:11.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:21:11.899 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:11.896+0000 7fefe85ce0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:21:11.901 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:11.898+0000 7fefe85ce0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:21:11.902 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:11.899+0000 7fefe85ce0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:21:12.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T09:21:12.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:21:12.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:21:12.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:21:12.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:21:12.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:12.303 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:21:12.303 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:21:12.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:12.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:21:12.672 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:12.669+0000 7fefe85ce0c0 -1 Falling back to public interface 2021-05-12T09:21:12.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:21:13.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:21:13.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:13.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:21:13.681 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:21:13.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:13.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:21:14.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:21:15.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:21:15.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:15.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:21:15.092 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:21:15.093 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:15.093 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:21:15.506 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:21:16.255 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:16.252+0000 7fefe85ce0c0 -1 osd.1 36 log_to_monitors {default=true} 2021-05-12T09:21:16.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:21:16.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:16.508 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:21:16.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:21:16.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:16.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:21:16.980 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/116868,v1:127.0.0.1:6811/116868] [v2:127.0.0.1:6812/116868,v1:127.0.0.1:6813/116868] exists,up cb3779ec-96c0-4e85-9aa4-c83b3e395169 2021-05-12T09:21:16.981 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:21:16.981 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:21:16.981 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:21:16.981 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-05-12T09:21:16.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=116868 2021-05-12T09:21:16.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-05-12T09:21:16.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:21:16.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:21:16.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T09:21:16.987 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:21:16.987 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-05-12T09:21:16.987 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:21:16.987 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:21:16.987 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:21:16.988 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:21:16.988 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-05-12T09:21:16.988 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-05-12T09:21:16.988 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:21:16.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:21:16.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:21:16.989 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:21:16.989 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:21:16.989 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:21:16.990 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:21:16.990 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:21:16.990 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:21:16.990 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:21:16.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:21:16.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:21:16.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:21:16.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:21:16.992 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:21:16.992 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:21:16.992 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:21:16.992 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:21:16.992 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:21:16.993 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-05-12T09:21:16.993 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T09:21:16.993 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:21:16.994 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:21:16.995 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-05-12T09:21:16.995 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T09:21:16.995 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:21:16.995 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:21:16.996 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:21:17.005 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:17.002+0000 7f4d0738b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:21:17.006 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:17.004+0000 7f4d0738b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:21:17.007 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:17.005+0000 7f4d0738b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:21:17.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T09:21:17.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:21:17.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:21:17.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:21:17.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:21:17.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:17.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:21:17.377 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:21:17.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:17.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:21:17.529 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:17.526+0000 7f4d0738b0c0 -1 Falling back to public interface 2021-05-12T09:21:17.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:21:18.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:21:18.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:18.756 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:21:18.756 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:21:18.757 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:18.757 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:21:19.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:21:20.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:21:20.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:20.145 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:21:20.146 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:21:20.146 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:20.147 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:21:20.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:21:20.565 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:20.563+0000 7f4d0738b0c0 -1 osd.2 38 log_to_monitors {default=true} 2021-05-12T09:21:21.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:21:21.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:21.538 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:21:21.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:21:21.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:21.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:21:22.223 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 50 up_thru 50 down_at 39 last_clean_interval [15,38) [v2:127.0.0.1:6818/117829,v1:127.0.0.1:6819/117829] [v2:127.0.0.1:6820/117829,v1:127.0.0.1:6821/117829] exists,up 5bffcfc9-40a9-4ed4-900b-d2358b9b5f65 2021-05-12T09:21:22.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:21:22.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:21:22.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:21:22.226 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-05-12T09:21:22.229 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=117829 2021-05-12T09:21:22.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-05-12T09:21:22.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:21:22.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:21:22.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=3 2021-05-12T09:21:22.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:21:22.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-05-12T09:21:22.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:21:22.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:21:22.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:21:22.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:21:22.233 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-05-12T09:21:22.233 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-05-12T09:21:22.233 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:21:22.233 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:21:22.233 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:21:22.234 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:21:22.234 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:21:22.234 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:21:22.234 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:21:22.235 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:21:22.235 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:21:22.235 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:21:22.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:21:22.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:21:22.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:21:22.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:21:22.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:21:22.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:21:22.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:21:22.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:21:22.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:21:22.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-05-12T09:21:22.238 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.3 2021-05-12T09:21:22.239 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.3 2021-05-12T09:21:22.240 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 3 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:21:22.240 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-05-12T09:21:22.240 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 3 = 3 ']' 2021-05-12T09:21:22.240 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:21:22.241 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:21:22.242 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:21:22.265 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:22.261+0000 7fe1974c00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:21:22.288 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:22.285+0000 7fe1974c00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:21:22.292 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:22.287+0000 7fe1974c00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:21:22.806 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:22.803+0000 7fe1974c00c0 -1 Falling back to public interface 2021-05-12T09:21:22.902 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 3 2021-05-12T09:21:22.902 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:21:22.902 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T09:21:22.902 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:21:22.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:21:22.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:22.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:21:22.904 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:21:22.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:22.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:21:23.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:21:24.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:21:24.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:24.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:21:24.303 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:21:24.303 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:24.303 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:21:24.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:21:25.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:21:25.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:25.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:21:25.681 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:21:25.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:25.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:21:26.067 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:21:26.716 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:21:26.715+0000 7fe1974c00c0 -1 osd.3 40 log_to_monitors {default=true} 2021-05-12T09:21:27.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:21:27.069 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:21:27.069 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:21:27.069 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:21:27.070 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:21:27.070 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:21:27.528 INFO:tasks.workunit.client.0.smithi028.stdout:osd.3 up in weight 1 up_from 54 up_thru 39 down_at 41 last_clean_interval [24,40) [v2:127.0.0.1:6826/118780,v1:127.0.0.1:6827/118780] [v2:127.0.0.1:6828/118780,v1:127.0.0.1:6829/118780] exists,up da2b28e2-81ee-4600-aaab-71063cad2c8c 2021-05-12T09:21:27.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:21:27.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:21:27.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:21:27.530 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-05-12T09:21:27.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=118780 2021-05-12T09:21:27.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2021-05-12T09:21:27.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:21:27.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:21:27.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:21:27.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:21:27.532 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T09:21:27.533 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:21:27.533 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:21:27.534 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:21:27.534 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:21:27.534 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:21:27.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:21:27.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:21:27.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:21:27.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:21:27.622 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:21:28.161 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:21:28.162 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:21:28.162 INFO:tasks.workunit.client.0.smithi028.stderr:2 2021-05-12T09:21:28.162 INFO:tasks.workunit.client.0.smithi028.stderr:3' 2021-05-12T09:21:28.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:21:28.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:21:28.163 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:21:28.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=184683593734 2021-05-12T09:21:28.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 184683593734 2021-05-12T09:21:28.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-184683593734' 2021-05-12T09:21:28.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:21:28.357 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:21:28.491 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=197568495621 2021-05-12T09:21:28.492 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 197568495621 2021-05-12T09:21:28.492 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-184683593734 1-197568495621' 2021-05-12T09:21:28.492 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:21:28.493 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:21:28.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=214748364804 2021-05-12T09:21:28.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 214748364804 2021-05-12T09:21:28.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-184683593734 1-197568495621 2-214748364804' 2021-05-12T09:21:28.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:21:28.626 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T09:21:28.758 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=231928233987 2021-05-12T09:21:28.758 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 231928233987 2021-05-12T09:21:28.759 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-184683593734 1-197568495621 2-214748364804 3-231928233987' 2021-05-12T09:21:28.759 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:21:28.760 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-184683593734 2021-05-12T09:21:28.760 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:21:28.760 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:21:28.761 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-184683593734 2021-05-12T09:21:28.761 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:21:28.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=184683593734 2021-05-12T09:21:28.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 184683593734' 2021-05-12T09:21:28.763 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 184683593734 2021-05-12T09:21:28.763 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:21:29.149 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 184683593732 -lt 184683593734 2021-05-12T09:21:29.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:21:30.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:21:30.151 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:21:30.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 184683593734 -lt 184683593734 2021-05-12T09:21:30.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:21:30.541 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-197568495621 2021-05-12T09:21:30.542 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:21:30.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:21:30.543 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-197568495621 2021-05-12T09:21:30.543 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:21:30.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=197568495621 2021-05-12T09:21:30.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 197568495621' 2021-05-12T09:21:30.545 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 197568495621 2021-05-12T09:21:30.545 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:21:30.937 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 197568495621 -lt 197568495621 2021-05-12T09:21:30.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:21:30.938 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-214748364804 2021-05-12T09:21:30.938 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:21:30.939 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:21:30.940 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-214748364804 2021-05-12T09:21:30.941 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:21:30.941 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=214748364804 2021-05-12T09:21:30.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 214748364804' 2021-05-12T09:21:30.942 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 214748364804 2021-05-12T09:21:30.943 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:21:31.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 214748364805 -lt 214748364804 2021-05-12T09:21:31.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:21:31.338 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-231928233987 2021-05-12T09:21:31.339 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:21:31.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T09:21:31.341 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-231928233987 2021-05-12T09:21:31.341 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:21:31.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=231928233987 2021-05-12T09:21:31.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 231928233987' 2021-05-12T09:21:31.343 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.3 seq 231928233987 2021-05-12T09:21:31.343 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T09:21:31.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 231928233987 -lt 231928233987 2021-05-12T09:21:31.729 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:21:31.729 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:21:31.730 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:21:32.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 17 == 0 2021-05-12T09:21:32.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:21:32.232 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:21:32.232 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:21:32.232 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:21:32.232 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:21:32.233 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:21:32.233 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:21:32.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=13 2021-05-12T09:21:32.597 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:21:32.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:21:32.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:21:33.102 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 13 = 17 2021-05-12T09:21:33.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 13 '!=' -1 2021-05-12T09:21:33.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T09:21:33.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=13 2021-05-12T09:21:33.104 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:21:33.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:21:33.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:21:33.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:21:33.204 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:21:33.204 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:21:33.204 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:21:33.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:21:33.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:21:33.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:21:33.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=17 2021-05-12T09:21:33.574 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:21:33.574 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:21:33.574 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:21:34.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 17 = 17 2021-05-12T09:21:34.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:21:34.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:21:34.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-05-12T09:21:34.133 INFO:tasks.workunit.client.0.smithi028.stdout:hints = 1 2021-05-12T09:21:34.134 INFO:tasks.workunit.client.0.smithi028.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-05-12T09:21:34.134 INFO:tasks.workunit.client.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_120226 2021-05-12T09:21:34.134 INFO:tasks.workunit.client.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T09:21:34.134 INFO:tasks.workunit.client.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2021-05-12T09:21:35.133 INFO:tasks.workunit.client.0.smithi028.stdout: 1 16 34 18 0.070304 0.0703125 0.174785 0.223391 2021-05-12T09:21:36.133 INFO:tasks.workunit.client.0.smithi028.stdout: 2 16 37 21 0.0410095 0.0117188 0.100545 0.21146 2021-05-12T09:21:37.133 INFO:tasks.workunit.client.0.smithi028.stdout: 3 16 37 21 0.0273397 0 - 0.21146 2021-05-12T09:21:38.134 INFO:tasks.workunit.client.0.smithi028.stdout: 4 16 45 29 0.0283165 0.015625 0.150893 0.707193 2021-05-12T09:21:39.134 INFO:tasks.workunit.client.0.smithi028.stdout: 5 16 66 50 0.0390575 0.0820312 0.259721 0.828903 2021-05-12T09:21:40.134 INFO:tasks.workunit.client.0.smithi028.stdout: 6 16 79 63 0.0410103 0.0507812 0.150166 0.695443 2021-05-12T09:21:41.134 INFO:tasks.workunit.client.0.smithi028.stdout: 7 16 79 63 0.0351517 0 - 0.695443 2021-05-12T09:21:42.134 INFO:tasks.workunit.client.0.smithi028.stdout: 8 16 123 107 0.0522393 0.0859375 0.245757 1.17688 2021-05-12T09:21:43.135 INFO:tasks.workunit.client.0.smithi028.stdout: 9 16 177 161 0.0698692 0.210938 0.196718 0.874584 2021-05-12T09:21:44.134 INFO:tasks.workunit.client.0.smithi028.stdout: 10 16 229 213 0.0831921 0.203125 0.340558 0.738053 2021-05-12T09:21:45.135 INFO:tasks.workunit.client.0.smithi028.stdout: 11 16 293 277 0.0983535 0.25 0.256325 0.628633 2021-05-12T09:21:46.135 INFO:tasks.workunit.client.0.smithi028.stdout: 12 16 352 336 0.10936 0.230469 0.175249 0.56489 2021-05-12T09:21:47.135 INFO:tasks.workunit.client.0.smithi028.stdout: 13 16 415 399 0.119876 0.246094 0.24349 0.517359 2021-05-12T09:21:48.135 INFO:tasks.workunit.client.0.smithi028.stdout: 14 16 475 459 0.128052 0.234375 0.283929 0.483007 2021-05-12T09:21:49.135 INFO:tasks.workunit.client.0.smithi028.stdout: 15 16 531 515 0.134097 0.21875 0.361668 0.461451 2021-05-12T09:21:50.136 INFO:tasks.workunit.client.0.smithi028.stdout: 16 16 590 574 0.140118 0.230469 0.264895 0.442836 2021-05-12T09:21:51.135 INFO:tasks.workunit.client.0.smithi028.stdout: 17 16 647 631 0.144972 0.222656 0.293697 0.427256 2021-05-12T09:21:52.136 INFO:tasks.workunit.client.0.smithi028.stdout: 18 16 706 690 0.149719 0.230469 0.234857 0.413935 2021-05-12T09:21:53.136 INFO:tasks.workunit.client.0.smithi028.stdout: 19 16 763 747 0.153557 0.222656 0.274462 0.404074 2021-05-12T09:21:54.136 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:21:54.134922+0000 min lat: 0.100545 max lat: 7.34366 avg lat: 0.39772 2021-05-12T09:21:54.136 INFO:tasks.workunit.client.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T09:21:54.136 INFO:tasks.workunit.client.0.smithi028.stdout: 20 16 814 798 0.155839 0.199219 0.455544 0.39772 2021-05-12T09:21:55.136 INFO:tasks.workunit.client.0.smithi028.stdout: 21 16 874 858 0.159577 0.234375 0.266415 0.389077 2021-05-12T09:21:56.136 INFO:tasks.workunit.client.0.smithi028.stdout: 22 16 926 910 0.161555 0.203125 0.280547 0.383978 2021-05-12T09:21:57.136 INFO:tasks.workunit.client.0.smithi028.stdout: 23 16 983 967 0.16421 0.222656 0.284982 0.378127 2021-05-12T09:21:58.136 INFO:tasks.workunit.client.0.smithi028.stdout: 24 16 1039 1023 0.166481 0.21875 0.226567 0.372528 2021-05-12T09:21:59.136 INFO:tasks.workunit.client.0.smithi028.stdout: 25 16 1098 1082 0.16904 0.230469 0.17142 0.367525 2021-05-12T09:22:00.137 INFO:tasks.workunit.client.0.smithi028.stdout: 26 16 1149 1133 0.1702 0.199219 0.301584 0.364406 2021-05-12T09:22:01.137 INFO:tasks.workunit.client.0.smithi028.stdout: 27 16 1207 1191 0.172286 0.226562 0.276863 0.361133 2021-05-12T09:22:02.137 INFO:tasks.workunit.client.0.smithi028.stdout: 28 16 1265 1249 0.174224 0.226562 0.251926 0.357651 2021-05-12T09:22:03.137 INFO:tasks.workunit.client.0.smithi028.stdout: 29 16 1313 1297 0.174681 0.1875 0.30807 0.355246 2021-05-12T09:22:04.137 INFO:tasks.workunit.client.0.smithi028.stdout: 30 16 1372 1356 0.176539 0.230469 0.256481 0.352172 2021-05-12T09:22:05.137 INFO:tasks.workunit.client.0.smithi028.stdout:Total time run: 30.1408 2021-05-12T09:22:05.138 INFO:tasks.workunit.client.0.smithi028.stdout:Total writes made: 1372 2021-05-12T09:22:05.138 INFO:tasks.workunit.client.0.smithi028.stdout:Write size: 4096 2021-05-12T09:22:05.138 INFO:tasks.workunit.client.0.smithi028.stdout:Object size: 4096 2021-05-12T09:22:05.139 INFO:tasks.workunit.client.0.smithi028.stdout:Bandwidth (MB/sec): 0.177811 2021-05-12T09:22:05.139 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev Bandwidth: 0.0849853 2021-05-12T09:22:05.139 INFO:tasks.workunit.client.0.smithi028.stdout:Max bandwidth (MB/sec): 0.25 2021-05-12T09:22:05.139 INFO:tasks.workunit.client.0.smithi028.stdout:Min bandwidth (MB/sec): 0 2021-05-12T09:22:05.140 INFO:tasks.workunit.client.0.smithi028.stdout:Average IOPS: 45 2021-05-12T09:22:05.140 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev IOPS: 21.7562 2021-05-12T09:22:05.140 INFO:tasks.workunit.client.0.smithi028.stdout:Max IOPS: 64 2021-05-12T09:22:05.140 INFO:tasks.workunit.client.0.smithi028.stdout:Min IOPS: 0 2021-05-12T09:22:05.140 INFO:tasks.workunit.client.0.smithi028.stdout:Average Latency(s): 0.35092 2021-05-12T09:22:05.141 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev Latency(s): 0.594731 2021-05-12T09:22:05.141 INFO:tasks.workunit.client.0.smithi028.stdout:Max latency(s): 7.34366 2021-05-12T09:22:05.141 INFO:tasks.workunit.client.0.smithi028.stdout:Min latency(s): 0.100545 2021-05-12T09:22:05.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 115881 2021-05-12T09:22:05.246 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2021-05-12T09:22:06.247 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 115881 2021-05-12T09:22:06.247 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (115881) - No such process 2021-05-12T09:22:06.247 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2021-05-12T09:22:06.703 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already down. 2021-05-12T09:22:06.715 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 116868 2021-05-12T09:22:06.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2021-05-12T09:22:07.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 116868 2021-05-12T09:22:07.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (116868) - No such process 2021-05-12T09:22:07.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2021-05-12T09:22:08.422 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already down. 2021-05-12T09:22:08.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 117829 2021-05-12T09:22:08.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2021-05-12T09:22:09.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 117829 2021-05-12T09:22:09.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (117829) - No such process 2021-05-12T09:22:09.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2021-05-12T09:22:10.136 INFO:tasks.workunit.client.0.smithi028.stderr:osd.2 is already down. 2021-05-12T09:22:10.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 118780 2021-05-12T09:22:10.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2021-05-12T09:22:11.149 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 118780 2021-05-12T09:22:11.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (118780) - No such process 2021-05-12T09:22:11.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2021-05-12T09:22:11.674 INFO:tasks.workunit.client.0.smithi028.stderr:osd.3 is already down. 2021-05-12T09:22:11.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T09:22:12.964 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:22:12.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2021-05-12T09:22:13.226 INFO:tasks.workunit.client.0.smithi028.stderr:512+0 records in 2021-05-12T09:22:13.226 INFO:tasks.workunit.client.0.smithi028.stderr:512+0 records out 2021-05-12T09:22:13.226 INFO:tasks.workunit.client.0.smithi028.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.25687 s, 2.1 GB/s 2021-05-12T09:22:13.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2021-05-12T09:22:13.236 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:22:17.760 INFO:tasks.workunit.client.0.smithi028.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2021-05-12T09:22:17.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T09:22:19.548 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:22:19.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T09:22:20.836 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:22:20.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2021-05-12T09:22:21.332 INFO:tasks.workunit.client.0.smithi028.stderr:1024+0 records in 2021-05-12T09:22:21.332 INFO:tasks.workunit.client.0.smithi028.stderr:1024+0 records out 2021-05-12T09:22:21.333 INFO:tasks.workunit.client.0.smithi028.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.4929 s, 2.2 GB/s 2021-05-12T09:22:21.333 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2021-05-12T09:22:21.343 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:22:29.119 INFO:tasks.workunit.client.0.smithi028.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2021-05-12T09:22:29.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2021-05-12T09:22:29.134 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:22:30.908 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T09:22:32.693 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:22:32.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T09:22:33.479 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:22:33.482 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2021-05-12T09:22:33.495 INFO:tasks.workunit.client.0.smithi028.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-05-12T09:22:33.495 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:22:34.269 INFO:tasks.workunit.client.0.smithi028.stdout:DB device added 2021-05-12T09:22:34.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2021-05-12T09:22:34.286 INFO:tasks.workunit.client.0.smithi028.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-05-12T09:22:34.287 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:22:35.557 INFO:tasks.workunit.client.0.smithi028.stdout:WAL device added 2021-05-12T09:22:35.559 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2021-05-12T09:22:35.572 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:22:37.346 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T09:22:39.130 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:22:39.132 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T09:22:40.417 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:22:40.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2021-05-12T09:22:40.433 INFO:tasks.workunit.client.0.smithi028.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-05-12T09:22:40.433 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:22:41.704 INFO:tasks.workunit.client.0.smithi028.stdout:WAL device added 2021-05-12T09:22:41.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2021-05-12T09:22:41.721 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:22:43.493 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2021-05-12T09:22:43.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T09:22:44.780 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:22:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-05-12T09:22:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:22:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:22:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T09:22:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:22:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-05-12T09:22:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:22:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:22:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:22:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:22:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-05-12T09:22:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-05-12T09:22:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:22:44.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:22:44.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:22:44.788 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:22:44.788 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:22:44.788 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:22:44.789 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:22:44.789 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:22:44.789 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:22:44.789 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:22:44.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:22:44.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:22:44.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:22:44.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:22:44.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:22:44.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:22:44.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:22:44.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:22:44.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:22:44.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-05-12T09:22:44.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T09:22:44.794 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:22:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:22:44.795 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-05-12T09:22:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T09:22:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:22:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:22:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:22:44.810 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:22:44.808+0000 7f495f0520c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:22:44.812 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:22:44.810+0000 7f495f0520c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:22:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:22:44.812+0000 7f495f0520c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:22:45.179 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T09:22:45.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:22:45.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:22:45.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:22:45.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:22:45.181 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:22:45.181 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:22:45.181 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:22:45.182 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:22:45.182 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:22:45.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:22:45.831 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:22:45.829+0000 7f495f0520c0 -1 Falling back to public interface 2021-05-12T09:22:46.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:22:46.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:22:46.580 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:22:46.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:22:46.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:22:46.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:22:46.972 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:22:47.973 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:22:47.973 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:22:47.974 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:22:47.974 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:22:47.975 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:22:47.975 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:22:48.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:22:49.367 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:22:49.367 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:22:49.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:22:49.368 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:22:49.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:22:49.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:22:49.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:22:50.425 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:22:50.423+0000 7f495f0520c0 -1 osd.0 56 log_to_monitors {default=true} 2021-05-12T09:22:50.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:22:50.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:22:50.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T09:22:50.763 INFO:tasks.workunit.client.0.smithi028.stdout:4 2021-05-12T09:22:50.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:22:50.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:22:51.179 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 67 up_thru 67 down_at 57 last_clean_interval [43,56) [v2:127.0.0.1:6802/126425,v1:127.0.0.1:6803/126425] [v2:127.0.0.1:6804/126425,v1:127.0.0.1:6805/126425] exists,up 4dec11cd-c5bb-4f63-9dcf-2ee80e825c62 2021-05-12T09:22:51.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:22:51.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:22:51.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:22:51.180 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-05-12T09:22:51.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=126425 2021-05-12T09:22:51.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-05-12T09:22:51.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:22:51.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:22:51.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T09:22:51.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:22:51.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-05-12T09:22:51.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:22:51.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:22:51.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:22:51.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:22:51.187 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-05-12T09:22:51.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-05-12T09:22:51.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:22:51.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:22:51.189 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:22:51.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:22:51.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:22:51.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:22:51.190 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:22:51.190 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:22:51.190 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:22:51.190 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:22:51.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:22:51.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:22:51.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:22:51.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:22:51.192 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:22:51.192 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:22:51.192 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:22:51.192 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:22:51.192 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:22:51.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-05-12T09:22:51.193 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:22:51.194 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T09:22:51.195 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:22:51.195 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-05-12T09:22:51.195 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T09:22:51.195 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:22:51.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:22:51.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:22:51.204 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:22:51.202+0000 7f65dcba00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:22:51.205 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:22:51.204+0000 7f65dcba00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:22:51.207 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:22:51.205+0000 7f65dcba00c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:22:51.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T09:22:51.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:22:51.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:22:51.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:22:51.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:22:51.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:22:51.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:22:51.946 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:22:51.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:22:51.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:22:51.974 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:22:52.225 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:22:52.223+0000 7f65dcba00c0 -1 Falling back to public interface 2021-05-12T09:22:52.976 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:22:52.976 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:22:52.976 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:22:52.977 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:22:52.977 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:22:52.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:22:53.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:22:54.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:22:54.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:22:54.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:22:54.369 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:22:54.370 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:22:54.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:22:54.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:22:55.756 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:22:55.757 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:22:55.757 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:22:55.757 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:22:55.758 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:22:55.758 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:22:56.142 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:22:56.694 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:22:56.691+0000 7f65dcba00c0 -1 osd.1 59 log_to_monitors {default=true} 2021-05-12T09:22:57.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:22:57.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:22:57.144 INFO:tasks.workunit.client.0.smithi028.stdout:4 2021-05-12T09:22:57.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T09:22:57.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:22:57.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:22:57.628 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 70 up_thru 70 down_at 60 last_clean_interval [46,59) [v2:127.0.0.1:6810/127422,v1:127.0.0.1:6811/127422] [v2:127.0.0.1:6812/127422,v1:127.0.0.1:6813/127422] exists,up cb3779ec-96c0-4e85-9aa4-c83b3e395169 2021-05-12T09:22:57.636 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:22:57.636 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:22:57.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:22:57.637 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-05-12T09:22:57.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=127422 2021-05-12T09:22:57.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-05-12T09:22:57.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:22:57.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:22:57.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T09:22:57.640 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:22:57.640 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-05-12T09:22:57.640 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:22:57.641 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:22:57.641 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:22:57.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:22:57.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-05-12T09:22:57.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-05-12T09:22:57.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:22:57.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:22:57.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:22:57.647 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:22:57.647 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:22:57.648 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:22:57.648 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:22:57.648 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:22:57.648 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:22:57.649 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:22:57.649 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:22:57.649 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:22:57.649 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:22:57.649 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:22:57.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:22:57.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:22:57.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:22:57.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:22:57.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:22:57.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-05-12T09:22:57.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T09:22:57.651 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:22:57.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:22:57.653 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-05-12T09:22:57.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T09:22:57.654 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:22:57.654 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:22:57.654 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:22:57.655 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:22:57.651+0000 7fce87fd70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:22:57.656 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:22:57.654+0000 7fce87fd70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:22:57.658 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:22:57.655+0000 7fce87fd70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:22:58.018 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T09:22:58.018 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:22:58.018 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:22:58.019 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:22:58.019 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:22:58.019 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:22:58.019 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:22:58.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:22:58.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:22:58.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:22:58.395 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:22:58.678 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:22:58.676+0000 7fce87fd70c0 -1 Falling back to public interface 2021-05-12T09:22:59.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:22:59.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:22:59.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:22:59.398 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:22:59.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:22:59.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:22:59.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:23:00.779 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:23:00.779 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:23:00.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:23:00.780 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:23:00.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:23:00.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:23:01.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:23:02.164 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:23:02.164 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:23:02.165 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:23:02.165 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:23:02.166 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:23:02.166 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:23:02.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:23:03.549 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:23:03.550 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:23:03.550 INFO:tasks.workunit.client.0.smithi028.stdout:4 2021-05-12T09:23:03.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T09:23:03.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:23:03.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:23:03.792 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:23:03.789+0000 7fce87fd70c0 -1 osd.2 62 log_to_monitors {default=true} 2021-05-12T09:23:03.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:23:04.965 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:23:04.966 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:23:04.966 INFO:tasks.workunit.client.0.smithi028.stdout:5 2021-05-12T09:23:04.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 5 2021-05-12T09:23:04.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:23:04.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:23:05.638 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 74 up_thru 74 down_at 63 last_clean_interval [50,62) [v2:127.0.0.1:6818/128419,v1:127.0.0.1:6819/128419] [v2:127.0.0.1:6820/128419,v1:127.0.0.1:6821/128419] exists,up 5bffcfc9-40a9-4ed4-900b-d2358b9b5f65 2021-05-12T09:23:05.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:23:05.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:23:05.640 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:23:05.644 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-05-12T09:23:05.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=128419 2021-05-12T09:23:05.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-05-12T09:23:05.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:23:05.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:23:05.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=3 2021-05-12T09:23:05.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:23:05.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-05-12T09:23:05.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:23:05.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:23:05.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:23:05.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:23:05.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-05-12T09:23:05.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-05-12T09:23:05.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:23:05.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:23:05.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:23:05.648 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:23:05.648 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:23:05.648 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:23:05.648 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:23:05.649 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:23:05.649 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:23:05.649 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:23:05.649 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:23:05.649 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:23:05.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:23:05.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:23:05.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:23:05.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:23:05.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:23:05.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:23:05.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:23:05.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-05-12T09:23:05.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.3 2021-05-12T09:23:05.652 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.3 2021-05-12T09:23:05.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 3 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:23:05.654 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-05-12T09:23:05.654 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 3 = 3 ']' 2021-05-12T09:23:05.654 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:23:05.654 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:23:05.655 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:23:05.675 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:23:05.673+0000 7f87951d10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:23:05.677 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:23:05.675+0000 7f87951d10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:23:05.679 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:23:05.676+0000 7f87951d10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:23:06.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 3 2021-05-12T09:23:06.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:23:06.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T09:23:06.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:23:06.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:23:06.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:23:06.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:23:06.048 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:23:06.049 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:23:06.049 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:23:06.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:23:06.452 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:23:06.448+0000 7f87951d10c0 -1 Falling back to public interface 2021-05-12T09:23:07.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:23:07.433 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:23:07.433 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:23:07.433 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:23:07.434 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:23:07.434 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:23:07.807 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:23:08.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:23:08.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:23:08.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:23:08.810 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:23:08.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:23:08.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:23:09.189 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:23:10.190 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:23:10.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:23:10.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:23:10.191 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:23:10.192 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:23:10.192 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:23:10.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:23:10.682 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:23:10.679+0000 7f87951d10c0 -1 osd.3 64 log_to_monitors {default=true} 2021-05-12T09:23:11.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:23:11.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:23:11.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T09:23:11.585 INFO:tasks.workunit.client.0.smithi028.stdout:4 2021-05-12T09:23:11.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:23:11.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:23:12.011 INFO:tasks.workunit.client.0.smithi028.stdout:osd.3 up in weight 1 up_from 77 up_thru 77 down_at 65 last_clean_interval [54,64) [v2:127.0.0.1:6826/129450,v1:127.0.0.1:6827/129450] [v2:127.0.0.1:6828/129450,v1:127.0.0.1:6829/129450] exists,up da2b28e2-81ee-4600-aaab-71063cad2c8c 2021-05-12T09:23:12.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:23:12.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:23:12.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:23:12.013 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-05-12T09:23:12.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=129450 2021-05-12T09:23:12.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-05-12T09:23:12.040 INFO:tasks.workunit.client.0.smithi028.stdout:hints = 1 2021-05-12T09:23:12.040 INFO:tasks.workunit.client.0.smithi028.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-05-12T09:23:12.040 INFO:tasks.workunit.client.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_130422 2021-05-12T09:23:12.040 INFO:tasks.workunit.client.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T09:23:12.041 INFO:tasks.workunit.client.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2021-05-12T09:23:13.040 INFO:tasks.workunit.client.0.smithi028.stdout: 1 16 33 17 0.0664 0.0664062 0.358125 0.368358 2021-05-12T09:23:14.040 INFO:tasks.workunit.client.0.smithi028.stdout: 2 16 50 34 0.0663977 0.0664062 0.167079 0.400993 2021-05-12T09:23:15.040 INFO:tasks.workunit.client.0.smithi028.stdout: 3 16 54 38 0.0494721 0.015625 0.201106 0.381268 2021-05-12T09:23:16.040 INFO:tasks.workunit.client.0.smithi028.stdout: 4 16 54 38 0.0371038 0 - 0.381268 2021-05-12T09:23:17.040 INFO:tasks.workunit.client.0.smithi028.stdout: 5 16 54 38 0.0296831 0 - 0.381268 2021-05-12T09:23:18.040 INFO:tasks.workunit.client.0.smithi028.stdout: 6 16 54 38 0.024736 0 - 0.381268 2021-05-12T09:23:19.041 INFO:tasks.workunit.client.0.smithi028.stdout: 7 16 54 38 0.0212023 0 - 0.381268 2021-05-12T09:23:20.041 INFO:tasks.workunit.client.0.smithi028.stdout: 8 16 54 38 0.018552 0 - 0.381268 2021-05-12T09:23:21.041 INFO:tasks.workunit.client.0.smithi028.stdout: 9 16 55 39 0.0169247 0.000651042 8.25402 0.583134 2021-05-12T09:23:22.041 INFO:tasks.workunit.client.0.smithi028.stdout: 10 16 78 62 0.0242154 0.0898438 9.64043 2.51935 2021-05-12T09:23:23.041 INFO:tasks.workunit.client.0.smithi028.stdout: 11 16 124 108 0.0383469 0.179688 0.432361 1.59579 2021-05-12T09:23:24.041 INFO:tasks.workunit.client.0.smithi028.stdout: 12 16 177 161 0.0524015 0.207031 0.219726 1.17447 2021-05-12T09:23:25.041 INFO:tasks.workunit.client.0.smithi028.stdout: 13 16 231 215 0.0645943 0.210938 0.268151 0.953266 2021-05-12T09:23:26.041 INFO:tasks.workunit.client.0.smithi028.stdout: 14 16 287 271 0.0756032 0.21875 0.340988 0.818902 2021-05-12T09:23:27.042 INFO:tasks.workunit.client.0.smithi028.stdout: 15 16 340 324 0.0843634 0.207031 0.210487 0.734077 2021-05-12T09:23:28.042 INFO:tasks.workunit.client.0.smithi028.stdout: 16 16 385 369 0.0900754 0.175781 0.410568 0.686506 2021-05-12T09:23:29.042 INFO:tasks.workunit.client.0.smithi028.stdout: 17 16 437 421 0.0967236 0.203125 0.358007 0.641582 2021-05-12T09:23:30.043 INFO:tasks.workunit.client.0.smithi028.stdout: 18 16 489 473 0.102633 0.203125 0.317666 0.605157 2021-05-12T09:23:31.042 INFO:tasks.workunit.client.0.smithi028.stdout: 19 16 539 523 0.10751 0.195312 0.298473 0.57652 2021-05-12T09:23:32.043 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:23:32.041556+0000 min lat: 0.150094 max lat: 9.8849 avg lat: 0.54539 2021-05-12T09:23:32.043 INFO:tasks.workunit.client.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T09:23:32.043 INFO:tasks.workunit.client.0.smithi028.stdout: 20 16 601 585 0.114242 0.242188 0.309726 0.54539 2021-05-12T09:23:33.042 INFO:tasks.workunit.client.0.smithi028.stdout: 21 16 650 634 0.117915 0.191406 0.385019 0.527668 2021-05-12T09:23:34.043 INFO:tasks.workunit.client.0.smithi028.stdout: 22 16 696 680 0.120722 0.179688 0.341234 0.51323 2021-05-12T09:23:35.043 INFO:tasks.workunit.client.0.smithi028.stdout: 23 16 751 735 0.124813 0.214844 0.343056 0.497558 2021-05-12T09:23:36.043 INFO:tasks.workunit.client.0.smithi028.stdout: 24 16 800 784 0.127586 0.191406 0.308475 0.485869 2021-05-12T09:23:37.043 INFO:tasks.workunit.client.0.smithi028.stdout: 25 16 854 838 0.130919 0.210938 0.216411 0.473709 2021-05-12T09:23:38.043 INFO:tasks.workunit.client.0.smithi028.stdout: 26 16 899 883 0.132643 0.175781 0.443374 0.465231 2021-05-12T09:23:39.043 INFO:tasks.workunit.client.0.smithi028.stdout: 27 16 958 942 0.136265 0.230469 0.284935 0.456665 2021-05-12T09:23:40.043 INFO:tasks.workunit.client.0.smithi028.stdout: 28 16 1013 997 0.139071 0.214844 0.296124 0.447334 2021-05-12T09:23:41.044 INFO:tasks.workunit.client.0.smithi028.stdout: 29 16 1069 1053 0.141817 0.21875 0.271428 0.438356 2021-05-12T09:23:42.044 INFO:tasks.workunit.client.0.smithi028.stdout: 30 16 1120 1104 0.14373 0.199219 0.326543 0.433002 2021-05-12T09:23:43.044 INFO:tasks.workunit.client.0.smithi028.stdout:Total time run: 30.2163 2021-05-12T09:23:43.044 INFO:tasks.workunit.client.0.smithi028.stdout:Total writes made: 1120 2021-05-12T09:23:43.045 INFO:tasks.workunit.client.0.smithi028.stdout:Write size: 4096 2021-05-12T09:23:43.045 INFO:tasks.workunit.client.0.smithi028.stdout:Object size: 4096 2021-05-12T09:23:43.045 INFO:tasks.workunit.client.0.smithi028.stdout:Bandwidth (MB/sec): 0.144789 2021-05-12T09:23:43.045 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev Bandwidth: 0.0895685 2021-05-12T09:23:43.045 INFO:tasks.workunit.client.0.smithi028.stdout:Max bandwidth (MB/sec): 0.242188 2021-05-12T09:23:43.046 INFO:tasks.workunit.client.0.smithi028.stdout:Min bandwidth (MB/sec): 0 2021-05-12T09:23:43.046 INFO:tasks.workunit.client.0.smithi028.stdout:Average IOPS: 37 2021-05-12T09:23:43.046 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev IOPS: 22.9387 2021-05-12T09:23:43.046 INFO:tasks.workunit.client.0.smithi028.stdout:Max IOPS: 62 2021-05-12T09:23:43.047 INFO:tasks.workunit.client.0.smithi028.stdout:Min IOPS: 0 2021-05-12T09:23:43.047 INFO:tasks.workunit.client.0.smithi028.stdout:Average Latency(s): 0.430741 2021-05-12T09:23:43.047 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev Latency(s): 1.00582 2021-05-12T09:23:43.047 INFO:tasks.workunit.client.0.smithi028.stdout:Max latency(s): 9.8849 2021-05-12T09:23:43.048 INFO:tasks.workunit.client.0.smithi028.stdout:Min latency(s): 0.150094 2021-05-12T09:23:43.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 126425 2021-05-12T09:23:43.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2021-05-12T09:23:44.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 126425 2021-05-12T09:23:44.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (126425) - No such process 2021-05-12T09:23:44.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2021-05-12T09:23:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already down. 2021-05-12T09:23:44.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 127422 2021-05-12T09:23:44.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2021-05-12T09:23:45.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 127422 2021-05-12T09:23:45.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (127422) - No such process 2021-05-12T09:23:45.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2021-05-12T09:23:46.312 INFO:tasks.workunit.client.0.smithi028.stderr:osd.1 is already down. 2021-05-12T09:23:46.322 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 128419 2021-05-12T09:23:46.322 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2021-05-12T09:23:47.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 128419 2021-05-12T09:23:47.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (128419) - No such process 2021-05-12T09:23:47.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2021-05-12T09:23:48.026 INFO:tasks.workunit.client.0.smithi028.stderr:osd.2 is already down. 2021-05-12T09:23:48.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 129450 2021-05-12T09:23:48.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2021-05-12T09:23:49.037 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 129450 2021-05-12T09:23:49.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (129450) - No such process 2021-05-12T09:23:49.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2021-05-12T09:23:49.675 INFO:tasks.workunit.client.0.smithi028.stderr:osd.3 is already down. 2021-05-12T09:23:49.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T09:23:51.467 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:23:51.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2021-05-12T09:23:51.962 INFO:tasks.workunit.client.0.smithi028.stderr:1024+0 records in 2021-05-12T09:23:51.963 INFO:tasks.workunit.client.0.smithi028.stderr:1024+0 records out 2021-05-12T09:23:51.963 INFO:tasks.workunit.client.0.smithi028.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.49165 s, 2.2 GB/s 2021-05-12T09:23:51.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2021-05-12T09:23:51.973 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:24:00.500 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2021-05-12T09:24:00.500 INFO:tasks.workunit.client.0.smithi028.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2021-05-12T09:24:00.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T09:24:02.287 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:24:02.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2021-05-12T09:24:02.547 INFO:tasks.workunit.client.0.smithi028.stderr:512+0 records in 2021-05-12T09:24:02.547 INFO:tasks.workunit.client.0.smithi028.stderr:512+0 records out 2021-05-12T09:24:02.548 INFO:tasks.workunit.client.0.smithi028.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.255359 s, 2.1 GB/s 2021-05-12T09:24:02.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2021-05-12T09:24:02.556 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:24:07.577 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2021-05-12T09:24:07.578 INFO:tasks.workunit.client.0.smithi028.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2021-05-12T09:24:07.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2021-05-12T09:24:07.631 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-05-12T09:24:09.414 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:24:09.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T09:24:11.204 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:24:11.207 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2021-05-12T09:24:11.698 INFO:tasks.workunit.client.0.smithi028.stderr:1024+0 records in 2021-05-12T09:24:11.699 INFO:tasks.workunit.client.0.smithi028.stderr:1024+0 records out 2021-05-12T09:24:11.699 INFO:tasks.workunit.client.0.smithi028.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.489485 s, 2.2 GB/s 2021-05-12T09:24:11.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2021-05-12T09:24:11.709 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:24:19.983 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-05-12T09:24:19.983 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2021-05-12T09:24:19.983 INFO:tasks.workunit.client.0.smithi028.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2021-05-12T09:24:19.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2021-05-12T09:24:20.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T09:24:21.361 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:24:21.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2021-05-12T09:24:21.377 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:24:22.646 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-05-12T09:24:22.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2021-05-12T09:24:22.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-05-12T09:24:23.532 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:24:23.534 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2021-05-12T09:24:25.320 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:24:25.322 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2021-05-12T09:24:25.820 INFO:tasks.workunit.client.0.smithi028.stderr:1024+0 records in 2021-05-12T09:24:25.820 INFO:tasks.workunit.client.0.smithi028.stderr:1024+0 records out 2021-05-12T09:24:25.820 INFO:tasks.workunit.client.0.smithi028.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.496045 s, 2.2 GB/s 2021-05-12T09:24:25.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2021-05-12T09:24:25.830 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:24:25.835 INFO:tasks.workunit.client.0.smithi028.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2021-05-12T09:24:25.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2021-05-12T09:24:25.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T09:24:27.618 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:24:27.620 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2021-05-12T09:24:34.664 INFO:tasks.workunit.client.0.smithi028.stderr:1024+0 records in 2021-05-12T09:24:34.665 INFO:tasks.workunit.client.0.smithi028.stderr:1024+0 records out 2021-05-12T09:24:34.665 INFO:tasks.workunit.client.0.smithi028.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.91197 s, 155 MB/s 2021-05-12T09:24:34.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2021-05-12T09:24:34.678 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:24:36.449 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2021-05-12T09:24:36.449 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2021-05-12T09:24:36.449 INFO:tasks.workunit.client.0.smithi028.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2021-05-12T09:24:36.451 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-05-12T09:24:37.736 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:24:37.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2021-05-12T09:24:38.233 INFO:tasks.workunit.client.0.smithi028.stderr:1024+0 records in 2021-05-12T09:24:38.233 INFO:tasks.workunit.client.0.smithi028.stderr:1024+0 records out 2021-05-12T09:24:38.234 INFO:tasks.workunit.client.0.smithi028.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.492648 s, 2.2 GB/s 2021-05-12T09:24:38.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2021-05-12T09:24:38.250 INFO:tasks.workunit.client.0.smithi028.stdout:inferring bluefs devices from bluestore path 2021-05-12T09:24:46.269 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2021-05-12T09:24:46.269 INFO:tasks.workunit.client.0.smithi028.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2021-05-12T09:24:46.269 INFO:tasks.workunit.client.0.smithi028.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2021-05-12T09:24:46.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-05-12T09:24:47.556 INFO:tasks.workunit.client.0.smithi028.stdout:fsck success 2021-05-12T09:24:47.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-05-12T09:24:47.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:24:47.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:24:47.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T09:24:47.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:24:47.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-05-12T09:24:47.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:24:47.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:24:47.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:24:47.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:24:47.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-05-12T09:24:47.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-05-12T09:24:47.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:24:47.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:24:47.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:24:47.565 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:24:47.566 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:24:47.566 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:24:47.566 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:24:47.566 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:24:47.567 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:24:47.567 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:24:47.569 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:24:47.569 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:24:47.569 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:24:47.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:24:47.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:24:47.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:24:47.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:24:47.571 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:24:47.571 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:24:47.571 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-05-12T09:24:47.572 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:24:47.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T09:24:47.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:24:47.574 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-05-12T09:24:47.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T09:24:47.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:24:47.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:24:47.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:24:47.586 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:24:47.584+0000 7fcde06570c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:24:47.588 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:24:47.586+0000 7fcde06570c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:24:47.589 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:24:47.587+0000 7fcde06570c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:24:47.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T09:24:47.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:24:47.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:24:47.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:24:47.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:24:47.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:24:47.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:24:47.948 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:24:47.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:24:47.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:24:48.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:24:48.607 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:24:48.604+0000 7fcde06570c0 -1 Falling back to public interface 2021-05-12T09:24:49.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:24:49.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:24:49.344 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:24:49.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:24:49.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:24:49.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:24:49.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:24:50.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:24:50.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:24:50.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:24:50.730 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:24:50.730 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:24:50.731 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:24:51.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:24:52.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:24:52.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:24:52.114 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:24:52.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:24:52.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:24:52.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:24:52.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:24:53.124 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:24:53.121+0000 7fcde06570c0 -1 osd.0 79 log_to_monitors {default=true} 2021-05-12T09:24:53.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:24:53.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:24:53.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T09:24:53.501 INFO:tasks.workunit.client.0.smithi028.stdout:4 2021-05-12T09:24:53.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:24:53.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:24:53.904 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 90 up_thru 90 down_at 80 last_clean_interval [67,79) [v2:127.0.0.1:6802/136707,v1:127.0.0.1:6803/136707] [v2:127.0.0.1:6804/136707,v1:127.0.0.1:6805/136707] exists,up 4dec11cd-c5bb-4f63-9dcf-2ee80e825c62 2021-05-12T09:24:53.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:24:53.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:24:53.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:24:53.906 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-05-12T09:24:53.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=136707 2021-05-12T09:24:53.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-05-12T09:24:53.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:24:53.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:24:53.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T09:24:53.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:24:53.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-05-12T09:24:53.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:24:53.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:24:53.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:24:53.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:24:53.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-05-12T09:24:53.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-05-12T09:24:53.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:24:53.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:24:53.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:24:53.915 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:24:53.915 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:24:53.915 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:24:53.916 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:24:53.916 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:24:53.916 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:24:53.916 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:24:53.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:24:53.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:24:53.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:24:53.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:24:53.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:24:53.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:24:53.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:24:53.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:24:53.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:24:53.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-05-12T09:24:53.920 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:24:53.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T09:24:53.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:24:53.921 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-05-12T09:24:53.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T09:24:53.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:24:53.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:24:53.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:24:53.929 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:24:53.927+0000 7f532bedd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:24:53.932 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:24:53.929+0000 7f532bedd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:24:53.933 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:24:53.930+0000 7f532bedd0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:24:54.306 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T09:24:54.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:24:54.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:24:54.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:24:54.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:24:54.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:24:54.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:24:54.308 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:24:54.309 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:24:54.309 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:24:54.453 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:24:54.451+0000 7f532bedd0c0 -1 Falling back to public interface 2021-05-12T09:24:54.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:24:55.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:24:55.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:24:55.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:24:55.710 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:24:55.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:24:55.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:24:56.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:24:57.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:24:57.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:24:57.099 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:24:57.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:24:57.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:24:57.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:24:57.488 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:24:57.871 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:24:57.868+0000 7f532bedd0c0 -1 osd.1 82 log_to_monitors {default=true} 2021-05-12T09:24:58.491 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:24:58.492 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:24:58.493 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:24:58.495 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:24:58.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:24:58.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:24:58.883 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 93 up_thru 93 down_at 83 last_clean_interval [70,82) [v2:127.0.0.1:6810/137704,v1:127.0.0.1:6811/137704] [v2:127.0.0.1:6812/137704,v1:127.0.0.1:6813/137704] exists,up cb3779ec-96c0-4e85-9aa4-c83b3e395169 2021-05-12T09:24:58.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:24:58.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:24:58.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:24:58.893 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-05-12T09:24:58.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=137704 2021-05-12T09:24:58.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-05-12T09:24:58.894 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:24:58.894 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:24:58.894 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T09:24:58.894 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:24:58.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-05-12T09:24:58.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:24:58.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:24:58.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:24:58.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:24:58.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-05-12T09:24:58.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-05-12T09:24:58.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:24:58.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:24:58.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:24:58.897 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:24:58.897 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:24:58.898 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:24:58.898 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:24:58.898 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:24:58.898 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:24:58.899 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:24:58.899 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:24:58.899 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:24:58.899 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:24:58.899 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:24:58.900 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:24:58.900 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:24:58.900 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:24:58.901 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:24:58.901 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:24:58.901 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-05-12T09:24:58.902 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:24:58.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T09:24:58.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:24:58.903 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-05-12T09:24:58.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T09:24:58.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:24:58.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:24:58.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:24:58.906 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:24:58.904+0000 7f18ffff10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:24:58.908 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:24:58.906+0000 7f18ffff10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:24:58.909 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:24:58.907+0000 7f18ffff10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:24:59.267 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T09:24:59.268 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:24:59.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:24:59.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:24:59.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:24:59.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:24:59.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:24:59.272 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:24:59.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:24:59.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:24:59.659 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:24:59.675 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:24:59.674+0000 7f18ffff10c0 -1 Falling back to public interface 2021-05-12T09:25:00.660 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:25:00.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:25:00.661 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:25:00.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:25:00.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:25:00.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:25:01.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:25:02.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:25:02.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:25:02.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:25:02.051 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:25:02.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:25:02.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:25:02.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:25:03.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:25:03.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:25:03.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:25:03.441 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:25:03.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:25:03.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:25:03.524 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:03.523+0000 7f18ffff10c0 -1 osd.2 85 log_to_monitors {default=true} 2021-05-12T09:25:03.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:25:04.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:25:04.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:25:04.837 INFO:tasks.workunit.client.0.smithi028.stdout:4 2021-05-12T09:25:04.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 4 2021-05-12T09:25:04.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:25:04.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:25:05.529 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 96 up_thru 96 down_at 86 last_clean_interval [74,85) [v2:127.0.0.1:6818/138655,v1:127.0.0.1:6819/138655] [v2:127.0.0.1:6820/138655,v1:127.0.0.1:6821/138655] exists,up 5bffcfc9-40a9-4ed4-900b-d2358b9b5f65 2021-05-12T09:25:05.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:25:05.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:25:05.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:25:05.531 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-05-12T09:25:05.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=138655 2021-05-12T09:25:05.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-05-12T09:25:05.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:25:05.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:25:05.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=3 2021-05-12T09:25:05.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:25:05.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-05-12T09:25:05.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-05-12T09:25:05.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:25:05.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:25:05.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:25:05.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-05-12T09:25:05.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-05-12T09:25:05.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:25:05.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:25:05.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-05-12T09:25:05.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:25:05.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:25:05.540 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:25:05.540 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:25:05.540 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:05.540 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:25:05.541 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:25:05.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' 2021-05-12T09:25:05.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:25:05.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-05-12T09:25:05.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-05-12T09:25:05.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:25:05.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:25:05.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:25:05.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:25:05.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:25:05.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-05-12T09:25:05.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.3 2021-05-12T09:25:05.543 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.3 2021-05-12T09:25:05.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 3 --fsid=ecc7095b-7872-4609-b610-e19ea5965c2d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94395/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:25:05.545 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-05-12T09:25:05.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 3 = 3 ']' 2021-05-12T09:25:05.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:25:05.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:25:05.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:25:05.555 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:05.554+0000 7f63231730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:25:05.558 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:05.556+0000 7f63231730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:25:05.559 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:05.557+0000 7f63231730c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:25:05.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 3 2021-05-12T09:25:05.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:25:05.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=3 2021-05-12T09:25:05.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:25:05.955 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:25:05.955 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:25:05.955 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:25:05.956 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:25:05.956 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:25:05.957 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:25:06.324 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:06.323+0000 7f63231730c0 -1 Falling back to public interface 2021-05-12T09:25:06.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:25:07.331 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:25:07.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:25:07.332 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:25:07.333 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:25:07.333 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:25:07.333 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:25:07.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:25:08.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:25:08.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:25:08.726 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:25:08.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:25:08.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:25:08.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:25:09.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:25:10.075 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:10.073+0000 7f63231730c0 -1 osd.3 87 log_to_monitors {default=true} 2021-05-12T09:25:10.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:25:10.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:25:10.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 3 2021-05-12T09:25:10.100 INFO:tasks.workunit.client.0.smithi028.stdout:3 2021-05-12T09:25:10.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:25:10.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.3 up' 2021-05-12T09:25:10.561 INFO:tasks.workunit.client.0.smithi028.stdout:osd.3 up in weight 1 up_from 100 up_thru 86 down_at 88 last_clean_interval [77,87) [v2:127.0.0.1:6826/139650,v1:127.0.0.1:6827/139650] [v2:127.0.0.1:6828/139650,v1:127.0.0.1:6829/139650] exists,up da2b28e2-81ee-4600-aaab-71063cad2c8c 2021-05-12T09:25:10.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:25:10.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:25:10.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:25:10.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-05-12T09:25:10.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=139650 2021-05-12T09:25:10.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-05-12T09:25:10.589 INFO:tasks.workunit.client.0.smithi028.stdout:hints = 1 2021-05-12T09:25:10.589 INFO:tasks.workunit.client.0.smithi028.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-05-12T09:25:10.589 INFO:tasks.workunit.client.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_140596 2021-05-12T09:25:10.590 INFO:tasks.workunit.client.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T09:25:10.590 INFO:tasks.workunit.client.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2021-05-12T09:25:11.589 INFO:tasks.workunit.client.0.smithi028.stdout: 1 16 19 3 0.0117179 0.0117188 0.248367 0.165114 2021-05-12T09:25:12.589 INFO:tasks.workunit.client.0.smithi028.stdout: 2 16 31 15 0.0292936 0.046875 0.0584828 0.380622 2021-05-12T09:25:13.589 INFO:tasks.workunit.client.0.smithi028.stdout: 3 16 31 15 0.0195288 0 - 0.380622 2021-05-12T09:25:14.589 INFO:tasks.workunit.client.0.smithi028.stdout: 4 16 31 15 0.0146465 0 - 0.380622 2021-05-12T09:25:15.590 INFO:tasks.workunit.client.0.smithi028.stdout: 5 16 36 20 0.0156229 0.00651042 0.113295 0.706581 2021-05-12T09:25:16.590 INFO:tasks.workunit.client.0.smithi028.stdout: 6 16 36 20 0.013019 0 - 0.706581 2021-05-12T09:25:17.590 INFO:tasks.workunit.client.0.smithi028.stdout: 7 16 36 20 0.0111591 0 - 0.706581 2021-05-12T09:25:18.590 INFO:tasks.workunit.client.0.smithi028.stdout: 8 16 36 20 0.00976425 0 - 0.706581 2021-05-12T09:25:19.590 INFO:tasks.workunit.client.0.smithi028.stdout: 9 16 36 20 0.00867935 0 - 0.706581 2021-05-12T09:25:20.590 INFO:tasks.workunit.client.0.smithi028.stdout: 10 16 36 20 0.00781144 0 - 0.706581 2021-05-12T09:25:21.590 INFO:tasks.workunit.client.0.smithi028.stdout: 11 16 56 40 0.0142026 0.0130208 0.176133 1.55815 2021-05-12T09:25:22.590 INFO:tasks.workunit.client.0.smithi028.stdout: 12 16 64 48 0.0156229 0.03125 0.108783 1.3208 2021-05-12T09:25:23.591 INFO:tasks.workunit.client.0.smithi028.stdout: 13 16 142 126 0.0378556 0.304688 0.176173 1.63428 2021-05-12T09:25:24.591 INFO:tasks.workunit.client.0.smithi028.stdout: 14 16 225 209 0.058307 0.324219 0.163817 1.06189 2021-05-12T09:25:25.591 INFO:tasks.workunit.client.0.smithi028.stdout: 15 16 306 290 0.0755108 0.316406 0.139713 0.821081 2021-05-12T09:25:26.591 INFO:tasks.workunit.client.0.smithi028.stdout: 16 16 376 360 0.0878791 0.273438 0.265514 0.704681 2021-05-12T09:25:27.591 INFO:tasks.workunit.client.0.smithi028.stdout: 17 16 455 439 0.10086 0.308594 0.219316 0.613418 2021-05-12T09:25:28.591 INFO:tasks.workunit.client.0.smithi028.stdout: 18 16 542 526 0.114134 0.339844 0.102163 0.544737 2021-05-12T09:25:29.591 INFO:tasks.workunit.client.0.smithi028.stdout: 19 16 626 610 0.125395 0.328125 0.212077 0.49541 2021-05-12T09:25:30.591 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:25:30.590624+0000 min lat: 0.0534998 max lat: 12.1679 avg lat: 0.457438 2021-05-12T09:25:30.592 INFO:tasks.workunit.client.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T09:25:30.592 INFO:tasks.workunit.client.0.smithi028.stdout: 20 16 712 696 0.13592 0.335938 0.180161 0.457438 2021-05-12T09:25:31.592 INFO:tasks.workunit.client.0.smithi028.stdout: 21 16 781 765 0.14228 0.269531 0.246884 0.434294 2021-05-12T09:25:32.592 INFO:tasks.workunit.client.0.smithi028.stdout: 22 16 857 841 0.149305 0.296875 0.138635 0.416973 2021-05-12T09:25:33.592 INFO:tasks.workunit.client.0.smithi028.stdout: 23 16 932 916 0.15555 0.292969 0.240868 0.400212 2021-05-12T09:25:34.593 INFO:tasks.workunit.client.0.smithi028.stdout: 24 16 1005 989 0.160948 0.285156 0.16075 0.386236 2021-05-12T09:25:35.592 INFO:tasks.workunit.client.0.smithi028.stdout: 25 16 1075 1059 0.165446 0.273438 0.312825 0.374762 2021-05-12T09:25:36.592 INFO:tasks.workunit.client.0.smithi028.stdout: 26 16 1160 1144 0.171852 0.332031 0.137504 0.362819 2021-05-12T09:25:37.592 INFO:tasks.workunit.client.0.smithi028.stdout: 27 16 1211 1195 0.172865 0.199219 0.20334 0.359808 2021-05-12T09:25:38.593 INFO:tasks.workunit.client.0.smithi028.stdout: 28 16 1285 1269 0.177013 0.289062 0.227255 0.351939 2021-05-12T09:25:39.593 INFO:tasks.workunit.client.0.smithi028.stdout: 29 16 1351 1335 0.179798 0.257812 0.43419 0.345449 2021-05-12T09:25:40.593 INFO:tasks.workunit.client.0.smithi028.stdout: 30 16 1427 1411 0.183699 0.296875 0.205603 0.338673 2021-05-12T09:25:41.593 INFO:tasks.workunit.client.0.smithi028.stdout:Total time run: 30.0728 2021-05-12T09:25:41.594 INFO:tasks.workunit.client.0.smithi028.stdout:Total writes made: 1427 2021-05-12T09:25:41.594 INFO:tasks.workunit.client.0.smithi028.stdout:Write size: 4096 2021-05-12T09:25:41.595 INFO:tasks.workunit.client.0.smithi028.stdout:Object size: 4096 2021-05-12T09:25:41.595 INFO:tasks.workunit.client.0.smithi028.stdout:Bandwidth (MB/sec): 0.185358 2021-05-12T09:25:41.595 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev Bandwidth: 0.145538 2021-05-12T09:25:41.595 INFO:tasks.workunit.client.0.smithi028.stdout:Max bandwidth (MB/sec): 0.339844 2021-05-12T09:25:41.596 INFO:tasks.workunit.client.0.smithi028.stdout:Min bandwidth (MB/sec): 0 2021-05-12T09:25:41.596 INFO:tasks.workunit.client.0.smithi028.stdout:Average IOPS: 47 2021-05-12T09:25:41.596 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev IOPS: 37.2987 2021-05-12T09:25:41.596 INFO:tasks.workunit.client.0.smithi028.stdout:Max IOPS: 87 2021-05-12T09:25:41.597 INFO:tasks.workunit.client.0.smithi028.stdout:Min IOPS: 0 2021-05-12T09:25:41.597 INFO:tasks.workunit.client.0.smithi028.stdout:Average Latency(s): 0.336865 2021-05-12T09:25:41.597 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev Latency(s): 1.13108 2021-05-12T09:25:41.597 INFO:tasks.workunit.client.0.smithi028.stdout:Max latency(s): 12.1679 2021-05-12T09:25:41.597 INFO:tasks.workunit.client.0.smithi028.stdout:Min latency(s): 0.0534998 2021-05-12T09:25:41.656 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2021-05-12T09:25:41.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:25:41.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:25:41.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:25:41.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:25:41.659 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T09:25:41.659 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:25:41.659 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:25:41.659 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:25:41.660 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:25:41.660 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:25:41.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:25:41.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:25:41.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:25:41.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:25:41.765 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:25:42.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:25:42.145 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:25:42.145 INFO:tasks.workunit.client.0.smithi028.stderr:2 2021-05-12T09:25:42.146 INFO:tasks.workunit.client.0.smithi028.stderr:3' 2021-05-12T09:25:42.146 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:25:42.146 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:25:42.147 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:25:42.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=386547056652 2021-05-12T09:25:42.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 386547056652 2021-05-12T09:25:42.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-386547056652' 2021-05-12T09:25:42.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:25:42.297 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:25:42.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=399431958539 2021-05-12T09:25:42.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 399431958539 2021-05-12T09:25:42.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-386547056652 1-399431958539' 2021-05-12T09:25:42.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:25:42.429 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:25:42.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=412316860426 2021-05-12T09:25:42.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 412316860426 2021-05-12T09:25:42.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-386547056652 1-399431958539 2-412316860426' 2021-05-12T09:25:42.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:25:42.561 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-05-12T09:25:42.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=429496729609 2021-05-12T09:25:42.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 429496729609 2021-05-12T09:25:42.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-386547056652 1-399431958539 2-412316860426 3-429496729609' 2021-05-12T09:25:42.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:25:42.694 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-386547056652 2021-05-12T09:25:42.694 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:25:42.695 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:25:42.695 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-386547056652 2021-05-12T09:25:42.696 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:25:42.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=386547056652 2021-05-12T09:25:42.697 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 386547056652' 2021-05-12T09:25:42.697 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 386547056652 2021-05-12T09:25:42.698 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:25:43.072 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 386547056650 -lt 386547056652 2021-05-12T09:25:43.073 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:25:44.074 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:25:44.075 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:25:44.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 386547056652 -lt 386547056652 2021-05-12T09:25:44.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:25:44.463 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-399431958539 2021-05-12T09:25:44.463 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:25:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:25:44.465 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-399431958539 2021-05-12T09:25:44.465 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:25:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=399431958539 2021-05-12T09:25:44.466 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 399431958539 2021-05-12T09:25:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 399431958539' 2021-05-12T09:25:44.467 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:25:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 399431958540 -lt 399431958539 2021-05-12T09:25:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:25:44.859 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-412316860426 2021-05-12T09:25:44.859 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:25:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:25:44.861 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-412316860426 2021-05-12T09:25:44.862 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:25:44.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=412316860426 2021-05-12T09:25:44.863 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 412316860426' 2021-05-12T09:25:44.863 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 412316860426 2021-05-12T09:25:44.863 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:25:45.257 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 412316860426 -lt 412316860426 2021-05-12T09:25:45.258 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:25:45.259 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-429496729609 2021-05-12T09:25:45.259 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:25:45.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=3 2021-05-12T09:25:45.260 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 3-429496729609 2021-05-12T09:25:45.261 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:25:45.261 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=429496729609 2021-05-12T09:25:45.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.3 seq 429496729609' 2021-05-12T09:25:45.262 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.3 seq 429496729609 2021-05-12T09:25:45.263 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 3 2021-05-12T09:25:45.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 429496729609 -lt 429496729609 2021-05-12T09:25:45.651 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:25:45.652 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:25:45.652 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:25:46.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 17 == 0 2021-05-12T09:25:46.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:25:46.150 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:25:46.150 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:25:46.150 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:25:46.151 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:25:46.151 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:25:46.151 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:25:46.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=17 2021-05-12T09:25:46.523 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:25:46.523 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:25:46.523 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:25:47.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 17 = 17 2021-05-12T09:25:47.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:25:47.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:25:47.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:25: run: teardown td/osd-bluefs-volume-ops 2021-05-12T09:25:47.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:25:47.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:25:47.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-05-12T09:25:47.022 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:25:47.023 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:25:47.023 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:25:47.023 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:25:47.023 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:25:47.130 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:25:47.131 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:25:47.132 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:25:47.132 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:25:47.133 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:25:47.133 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:25:47.133 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:25:47.134 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:25:47.135 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:25:47.135 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:25:47.136 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:25:47.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:25:47.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:25:47.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-05-12T09:25:47.508 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:25:47.509 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:47.509 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:25:47.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94395 2021-05-12T09:25:47.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:25:47.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:25:47.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T09:25:47.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-bluefs-volume-ops 0 2021-05-12T09:25:47.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-05-12T09:25:47.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T09:25:47.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-05-12T09:25:47.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:25:47.512 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:25:47.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:25:47.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:25:47.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:25:47.514 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:25:47.514 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:25:47.514 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:25:47.515 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:25:47.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:25:47.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:25:47.516 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:25:47.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:25:47.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:25:47.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:25:47.518 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:25:47.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:25:47.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T09:25:47.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-05-12T09:25:47.521 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:25:47.521 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:47.521 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94395 2021-05-12T09:25:47.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94395 2021-05-12T09:25:47.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:25:47.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:25:47.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T09:25:47.523 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T09:25:47.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T09:25:47.594 INFO:tasks.workunit:Running workunit osd/osd-config.sh... 2021-05-12T09:25:47.595 DEBUG:teuthology.orchestra.run.smithi028:workunit test osd/osd-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2021-05-12T09:25:47.655 INFO:tasks.workunit.client.0.smithi028.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T09:25:47.663 INFO:tasks.workunit.client.0.smithi028.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T09:25:47.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:25:47.664 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:25:47.664 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T09:25:47.664 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T09:25:47.664 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T09:25:47.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T09:25:47.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T09:25:47.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T09:25:47.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-config 2021-05-12T09:25:47.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:22: run: local dir=td/osd-config 2021-05-12T09:25:47.666 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:23: run: shift 2021-05-12T09:25:47.666 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: export CEPH_MON=127.0.0.1:7100 2021-05-12T09:25:47.666 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: CEPH_MON=127.0.0.1:7100 2021-05-12T09:25:47.666 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:26: run: export CEPH_ARGS 2021-05-12T09:25:47.667 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: uuidgen 2021-05-12T09:25:47.667 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: CEPH_ARGS+='--fsid=d9767aa3-0a91-4cbc-9411-d2cdd97ce010 --auth-supported=none ' 2021-05-12T09:25:47.667 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2021-05-12T09:25:47.667 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: set 2021-05-12T09:25:47.667 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T09:25:47.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: local 'funcs=TEST_config_init 2021-05-12T09:25:47.671 INFO:tasks.workunit.client.0.smithi028.stderr:TEST_config_track 2021-05-12T09:25:47.671 INFO:tasks.workunit.client.0.smithi028.stderr:TEST_default_adjustment' 2021-05-12T09:25:47.671 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-05-12T09:25:47.671 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-05-12T09:25:47.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-05-12T09:25:47.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-05-12T09:25:47.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T09:25:47.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:25:47.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T09:25:47.673 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:25:47.673 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:25:47.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:25:47.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:25:47.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:25:47.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:25:47.674 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:25:47.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:25:47.674 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:25:47.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:25:47.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:25:47.675 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:25:47.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:25:47.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:25:47.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:25:47.676 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:25:47.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:25:47.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:25:47.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T09:25:47.680 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:25:47.680 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:47.680 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:25:47.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141154 2021-05-12T09:25:47.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:25:47.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:25:47.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-05-12T09:25:47.682 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:25:47.682 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:47.682 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:25:47.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.141154 2021-05-12T09:25:47.682 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:25:47.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T09:25:47.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T09:25:47.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:25:47.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-05-12T09:25:47.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_init td/osd-config 2021-05-12T09:25:47.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:39: TEST_config_init: local dir=td/osd-config 2021-05-12T09:25:47.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:41: TEST_config_init: run_mon td/osd-config a 2021-05-12T09:25:47.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2021-05-12T09:25:47.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:25:47.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:25:47.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:25:47.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2021-05-12T09:25:47.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2021-05-12T09:25:48.032 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:25:48.032 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:25:48.033 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:25:48.033 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:25:48.033 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:48.033 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:25:48.033 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141154/$cluster-$name.asok' 2021-05-12T09:25:48.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.141154/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:25:48.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:25:48.277 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:25:48.278 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:25:48.278 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:25:48.278 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:25:48.278 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:25:48.279 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:25:48.279 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:25:48.279 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:25:48.279 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:25:48.280 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:48.280 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:25:48.280 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141154/ceph-mon.a.asok 2021-05-12T09:25:48.280 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:25:48.281 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.141154/ceph-mon.a.asok config get fsid 2021-05-12T09:25:48.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:25:48.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:25:48.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:25:48.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:25:48.381 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:25:48.381 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:25:48.381 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:25:48.381 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:25:48.382 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:25:48.382 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:48.382 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:25:48.382 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141154/ceph-mon.a.asok 2021-05-12T09:25:48.382 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:25:48.383 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.141154/ceph-mon.a.asok config get mon_host 2021-05-12T09:25:48.477 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:42: TEST_config_init: run_mgr td/osd-config x 2021-05-12T09:25:48.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2021-05-12T09:25:48.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:25:48.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:25:48.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:25:48.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2021-05-12T09:25:48.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:25:49.356 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:25:49.357 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:25:49.358 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:25:49.358 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:25:49.358 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:49.358 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:25:49.359 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141154/$cluster-$name.asok' 2021-05-12T09:25:49.359 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:25:49.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.141154/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:25:49.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:43: TEST_config_init: local stale=1000 2021-05-12T09:25:49.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:44: TEST_config_init: local cache=500 2021-05-12T09:25:49.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:45: TEST_config_init: run_osd td/osd-config 0 --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2021-05-12T09:25:49.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2021-05-12T09:25:49.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:25:49.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:25:49.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:25:49.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2021-05-12T09:25:49.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d9767aa3-0a91-4cbc-9411-d2cdd97ce010 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2021-05-12T09:25:49.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:25:49.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:25:49.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:25:49.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2021-05-12T09:25:49.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2021-05-12T09:25:49.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:25:49.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:25:49.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2021-05-12T09:25:49.387 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:25:49.387 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:25:49.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:25:49.388 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:25:49.388 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:49.388 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:25:49.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141154/$cluster-$name.asok' 2021-05-12T09:25:49.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.141154/$cluster-$name.asok' 2021-05-12T09:25:49.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:25:49.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:25:49.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:25:49.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2021-05-12T09:25:49.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2021-05-12T09:25:49.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:25:49.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:25:49.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:25:49.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:25:49.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+='--osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000' 2021-05-12T09:25:49.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-config/0 2021-05-12T09:25:49.391 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:25:49.392 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=d53d0549-e330-4171-9bfa-4389181cc89e 2021-05-12T09:25:49.392 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 d53d0549-e330-4171-9bfa-4389181cc89e 2021-05-12T09:25:49.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 d53d0549-e330-4171-9bfa-4389181cc89e' 2021-05-12T09:25:49.393 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:25:49.395 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAdn5tgJg55FxAAba0qmpJP6w/Srtht2/kXkg== 2021-05-12T09:25:49.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAdn5tgJg55FxAAba0qmpJP6w/Srtht2/kXkg=="}' 2021-05-12T09:25:49.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new d53d0549-e330-4171-9bfa-4389181cc89e -i td/osd-config/0/new.json 2021-05-12T09:25:49.709 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:25:49.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-config/0/new.json 2021-05-12T09:25:49.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=d9767aa3-0a91-4cbc-9411-d2cdd97ce010 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.141154/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 --mkfs --key AQAdn5tgJg55FxAAba0qmpJP6w/Srtht2/kXkg== --osd-uuid d53d0549-e330-4171-9bfa-4389181cc89e 2021-05-12T09:25:49.745 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:49.743+0000 7f7996bbe0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:25:49.746 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:49.745+0000 7f7996bbe0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:25:49.748 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:49.746+0000 7f7996bbe0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:25:49.749 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:49.747+0000 7f7996bbe0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T09:25:49.749 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:49.747+0000 7f7996bbe0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T09:25:49.749 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:49.747+0000 7f7996bbe0c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T09:25:49.790 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:49.788+0000 7f7996bbe0c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2021-05-12T09:25:54.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-config/0/keyring 2021-05-12T09:25:54.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:25:54.374 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:25:54.374 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:25:54.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:25:54.877 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:25:54.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:25:54.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=d9767aa3-0a91-4cbc-9411-d2cdd97ce010 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.141154/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2021-05-12T09:25:54.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:25:54.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:25:54.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:25:54.898 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:54.895+0000 7f80364c30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:25:54.900 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:54.897+0000 7f80364c30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:25:54.901 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:54.899+0000 7f80364c30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:25:55.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:25:55.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:25:55.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:25:55.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:25:55.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:25:55.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:25:55.257 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:25:55.257 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:25:55.258 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:25:55.258 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:25:55.420 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:55.418+0000 7f80364c30c0 -1 Falling back to public interface 2021-05-12T09:25:55.655 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:25:56.656 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:25:56.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:25:56.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:25:56.658 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:25:56.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:25:56.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:25:57.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:25:57.607 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:25:57.604+0000 7f80364c30c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:25:58.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:25:58.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:25:58.048 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:25:58.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:25:58.049 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:25:58.049 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:25:58.441 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/142274,v1:127.0.0.1:6803/142274] [v2:127.0.0.1:6804/142274,v1:127.0.0.1:6805/142274] exists,up d53d0549-e330-4171-9bfa-4389181cc89e 2021-05-12T09:25:58.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:25:58.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:25:58.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:25:58.444 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: get_asok_path osd.0 2021-05-12T09:25:58.444 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T09:25:58.445 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T09:25:58.445 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:25:58.445 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:58.445 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:25:58.446 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141154/ceph-osd.0.asok 2021-05-12T09:25:58.446 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: CEPH_ARGS= 2021-05-12T09:25:58.446 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: ceph --admin-daemon /tmp/ceph-asok.141154/ceph-osd.0.asok log flush 2021-05-12T09:25:58.541 INFO:tasks.workunit.client.0.smithi028.stdout:{} 2021-05-12T09:25:58.549 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:50: TEST_config_init: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-05-12T09:25:58.550 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:25:57.606+0000 7f802dc4c700 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2021-05-12T09:25:58.551 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:25:57.889+0000 7f80364c30c0 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2021-05-12T09:25:58.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-05-12T09:25:58.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T09:25:58.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:25:58.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T09:25:58.553 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:25:58.553 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:25:58.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:25:58.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:25:58.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:25:58.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:25:58.659 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:25:58.659 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:25:58.659 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:25:58.660 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:25:58.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:25:58.661 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:25:58.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:25:58.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:25:58.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:25:58.662 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:25:58.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:25:58.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:25:58.664 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T09:25:58.666 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:25:58.666 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:58.666 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:25:58.667 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141154 2021-05-12T09:25:58.668 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:25:58.669 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:25:58.669 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-05-12T09:25:58.669 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-05-12T09:25:58.669 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-05-12T09:25:58.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-05-12T09:25:58.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T09:25:58.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:25:58.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T09:25:58.670 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:25:58.671 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:25:58.671 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:25:58.671 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:25:58.671 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:25:58.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:25:58.673 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:25:58.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:25:58.673 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:25:58.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:25:58.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:25:58.675 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:25:58.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:25:58.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:25:58.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:25:58.677 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:25:58.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:25:58.678 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:25:58.678 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T09:25:58.678 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:25:58.679 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:58.679 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:25:58.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141154 2021-05-12T09:25:58.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:25:58.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:25:58.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-05-12T09:25:58.682 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:25:58.682 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:58.682 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:25:58.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.141154 2021-05-12T09:25:58.683 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:25:58.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T09:25:58.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:25:58.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-05-12T09:25:58.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_track td/osd-config 2021-05-12T09:25:58.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:54: TEST_config_track: local dir=td/osd-config 2021-05-12T09:25:58.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:56: TEST_config_track: run_mon td/osd-config a 2021-05-12T09:25:58.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2021-05-12T09:25:58.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:25:58.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:25:58.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:25:58.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2021-05-12T09:25:58.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2021-05-12T09:25:59.042 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:25:59.042 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:25:59.043 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:25:59.043 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:25:59.043 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:59.043 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:25:59.043 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141154/$cluster-$name.asok' 2021-05-12T09:25:59.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.141154/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:25:59.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:25:59.319 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:25:59.319 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:25:59.320 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:25:59.320 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:25:59.320 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:25:59.320 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:25:59.320 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:25:59.321 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:25:59.321 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:25:59.321 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:59.321 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:25:59.322 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141154/ceph-mon.a.asok 2021-05-12T09:25:59.322 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:25:59.323 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.141154/ceph-mon.a.asok config get fsid 2021-05-12T09:25:59.423 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:25:59.423 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:25:59.424 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:25:59.424 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:25:59.424 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:25:59.424 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:25:59.424 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:25:59.425 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:25:59.425 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:25:59.425 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:25:59.425 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:25:59.426 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141154/ceph-mon.a.asok 2021-05-12T09:25:59.426 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:25:59.426 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.141154/ceph-mon.a.asok config get mon_host 2021-05-12T09:25:59.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:57: TEST_config_track: run_mgr td/osd-config x 2021-05-12T09:25:59.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2021-05-12T09:25:59.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:25:59.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:25:59.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:25:59.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2021-05-12T09:25:59.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:26:00.073 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:26:00.073 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:26:00.073 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:26:00.074 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:26:00.074 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:00.074 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:26:00.074 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141154/$cluster-$name.asok' 2021-05-12T09:26:00.075 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:26:00.075 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.141154/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:26:00.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:58: TEST_config_track: run_osd td/osd-config 0 2021-05-12T09:26:00.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2021-05-12T09:26:00.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:26:00.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:26:00.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:26:00.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2021-05-12T09:26:00.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d9767aa3-0a91-4cbc-9411-d2cdd97ce010 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2021-05-12T09:26:00.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:26:00.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:26:00.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:26:00.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2021-05-12T09:26:00.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2021-05-12T09:26:00.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:26:00.102 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:26:00.102 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2021-05-12T09:26:00.102 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:26:00.102 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:26:00.103 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:26:00.103 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:26:00.103 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:00.103 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:26:00.103 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141154/$cluster-$name.asok' 2021-05-12T09:26:00.104 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.141154/$cluster-$name.asok' 2021-05-12T09:26:00.104 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:26:00.104 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:26:00.104 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:26:00.104 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2021-05-12T09:26:00.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2021-05-12T09:26:00.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:26:00.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:26:00.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:26:00.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:26:00.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:26:00.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-config/0 2021-05-12T09:26:00.106 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:26:00.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=2444febe-01e9-4b51-8bf6-1ea995f1239a 2021-05-12T09:26:00.107 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 2444febe-01e9-4b51-8bf6-1ea995f1239a' 2021-05-12T09:26:00.107 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 2444febe-01e9-4b51-8bf6-1ea995f1239a 2021-05-12T09:26:00.107 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:26:00.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAon5tgO7SRBhAA1YbDerk8ByVwoahddwPC8A== 2021-05-12T09:26:00.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAon5tgO7SRBhAA1YbDerk8ByVwoahddwPC8A=="}' 2021-05-12T09:26:00.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 2444febe-01e9-4b51-8bf6-1ea995f1239a -i td/osd-config/0/new.json 2021-05-12T09:26:00.426 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:26:00.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-config/0/new.json 2021-05-12T09:26:00.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=d9767aa3-0a91-4cbc-9411-d2cdd97ce010 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.141154/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAon5tgO7SRBhAA1YbDerk8ByVwoahddwPC8A== --osd-uuid 2444febe-01e9-4b51-8bf6-1ea995f1239a 2021-05-12T09:26:00.460 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:00.457+0000 7f36e84450c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:00.461 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:00.459+0000 7f36e84450c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:00.463 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:00.460+0000 7f36e84450c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:00.464 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:00.461+0000 7f36e84450c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T09:26:00.464 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:00.461+0000 7f36e84450c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T09:26:00.464 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:00.461+0000 7f36e84450c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-05-12T09:26:00.499 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:00.496+0000 7f36e84450c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2021-05-12T09:26:05.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-config/0/keyring 2021-05-12T09:26:05.141 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:26:05.141 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:26:05.142 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:26:05.142 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:26:05.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:26:05.650 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:26:05.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=d9767aa3-0a91-4cbc-9411-d2cdd97ce010 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.141154/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:26:05.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:26:05.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:26:05.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:26:05.670 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:05.667+0000 7f79a61560c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:05.689 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:05.686+0000 7f79a61560c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:05.690 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:05.687+0000 7f79a61560c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:06.029 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:26:06.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:26:06.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:26:06.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:26:06.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:26:06.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:06.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:26:06.033 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:26:06.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:06.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:26:06.206 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:06.204+0000 7f79a61560c0 -1 Falling back to public interface 2021-05-12T09:26:06.415 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:26:07.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:26:07.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:07.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:26:07.418 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:26:07.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:07.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:26:07.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:26:08.408 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:08.405+0000 7f79a61560c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:26:08.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:26:08.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:08.805 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:26:08.806 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:26:08.806 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:08.807 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:26:09.194 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/143965,v1:127.0.0.1:6803/143965] [v2:127.0.0.1:6804/143965,v1:127.0.0.1:6805/143965] exists,up 2444febe-01e9-4b51-8bf6-1ea995f1239a 2021-05-12T09:26:09.195 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:26:09.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:26:09.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:26:09.196 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: CEPH_ARGS= 2021-05-12T09:26:09.197 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: ceph-conf --show-config-value osd_map_cache_size 2021-05-12T09:26:09.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: local osd_map_cache_size=50 2021-05-12T09:26:09.206 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: CEPH_ARGS= 2021-05-12T09:26:09.206 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: ceph-conf --show-config-value osd_pg_epoch_persisted_max_stale 2021-05-12T09:26:09.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: local osd_pg_epoch_persisted_max_stale=40 2021-05-12T09:26:09.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:68: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-05-12T09:26:09.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:69: TEST_config_track: local stale=100 2021-05-12T09:26:09.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:70: TEST_config_track: ceph tell osd.0 injectargs '--osd-pg-epoch-persisted-max-stale 100' 2021-05-12T09:26:09.344 INFO:tasks.workunit.client.0.smithi028.stdout:{} 2021-05-12T09:26:09.355 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: get_asok_path osd.0 2021-05-12T09:26:09.355 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T09:26:09.355 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T09:26:09.356 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:26:09.356 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:09.356 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:26:09.356 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141154/ceph-osd.0.asok 2021-05-12T09:26:09.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: CEPH_ARGS= 2021-05-12T09:26:09.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.141154/ceph-osd.0.asok log flush 2021-05-12T09:26:09.453 INFO:tasks.workunit.client.0.smithi028.stdout:{} 2021-05-12T09:26:09.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:72: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-05-12T09:26:09.463 INFO:tasks.workunit.client.0.smithi028.stdout:2021-05-12T09:26:09.327+0000 7f799f8e3700 0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2021-05-12T09:26:09.464 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:73: TEST_config_track: rm td/osd-config/osd.0.log 2021-05-12T09:26:09.465 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: get_asok_path osd.0 2021-05-12T09:26:09.466 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T09:26:09.467 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T09:26:09.467 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:26:09.467 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:09.467 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:26:09.468 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141154/ceph-osd.0.asok 2021-05-12T09:26:09.468 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: CEPH_ARGS= 2021-05-12T09:26:09.468 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.141154/ceph-osd.0.asok log reopen 2021-05-12T09:26:09.563 INFO:tasks.workunit.client.0.smithi028.stdout:{} 2021-05-12T09:26:09.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-05-12T09:26:09.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T09:26:09.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:26:09.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T09:26:09.574 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:26:09.574 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:26:09.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:26:09.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:26:09.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:26:09.678 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:26:09.679 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:26:09.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:26:09.680 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:26:09.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:26:09.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:26:09.681 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:26:09.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:09.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:26:09.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:26:09.684 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:09.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:26:09.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:26:09.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T09:26:09.687 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:26:09.688 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:09.688 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:26:09.688 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141154 2021-05-12T09:26:09.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:26:09.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:26:09.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-05-12T09:26:09.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-05-12T09:26:09.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-05-12T09:26:09.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-05-12T09:26:09.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T09:26:09.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:26:09.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T09:26:09.693 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:26:09.693 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:26:09.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:26:09.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:26:09.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:26:09.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:26:09.694 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:26:09.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:26:09.696 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:26:09.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:26:09.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:26:09.697 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:26:09.697 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:09.697 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:26:09.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:26:09.698 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:09.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:26:09.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:26:09.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T09:26:09.701 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:26:09.701 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:09.701 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:26:09.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141154 2021-05-12T09:26:09.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:26:09.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:26:09.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-05-12T09:26:09.702 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:26:09.703 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:09.703 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:26:09.703 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.141154 2021-05-12T09:26:09.703 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:26:09.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T09:26:09.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:26:09.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-05-12T09:26:09.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_default_adjustment td/osd-config 2021-05-12T09:26:09.705 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin 2021-05-12T09:26:09.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: a= 2021-05-12T09:26:09.714 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin default 2021-05-12T09:26:09.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: b=default 2021-05-12T09:26:09.727 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin arg 2021-05-12T09:26:09.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: c=arg 2021-05-12T09:26:09.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:81: TEST_default_adjustment: '[' '' '!=' default ']' 2021-05-12T09:26:09.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:82: TEST_default_adjustment: '[' default = default ']' 2021-05-12T09:26:09.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:83: TEST_default_adjustment: '[' arg = arg ']' 2021-05-12T09:26:09.743 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file 2021-05-12T09:26:09.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: a=true 2021-05-12T09:26:09.755 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false 2021-05-12T09:26:09.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: b=false 2021-05-12T09:26:09.769 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false --log-to-file 2021-05-12T09:26:09.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: c=true 2021-05-12T09:26:09.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:88: TEST_default_adjustment: '[' true = true ']' 2021-05-12T09:26:09.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:89: TEST_default_adjustment: '[' false = false ']' 2021-05-12T09:26:09.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:90: TEST_default_adjustment: '[' true = true ']' 2021-05-12T09:26:09.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-05-12T09:26:09.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T09:26:09.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:26:09.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T09:26:09.785 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:26:09.785 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:26:09.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:26:09.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:26:09.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:26:09.787 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:26:09.788 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:26:09.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:26:09.788 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:26:09.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:26:09.789 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:26:09.789 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:26:09.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:09.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:26:09.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:26:09.792 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:09.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:26:09.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:26:09.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T09:26:09.793 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:26:09.793 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:09.793 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:26:09.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141154 2021-05-12T09:26:09.795 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:26:09.795 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:26:09.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T09:26:09.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-config 0 2021-05-12T09:26:09.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-05-12T09:26:09.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T09:26:09.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-05-12T09:26:09.797 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:26:09.797 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:26:09.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:26:09.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:26:09.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:26:09.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:26:09.798 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:26:09.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:26:09.799 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:26:09.800 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:26:09.800 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:26:09.800 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:26:09.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:09.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:26:09.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:26:09.801 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:09.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:26:09.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T09:26:09.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-05-12T09:26:09.804 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:26:09.804 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:09.804 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141154 2021-05-12T09:26:09.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141154 2021-05-12T09:26:09.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:26:09.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:26:09.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T09:26:09.805 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T09:26:09.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T09:26:09.868 INFO:tasks.workunit:Running workunit osd/osd-copy-from.sh... 2021-05-12T09:26:09.868 DEBUG:teuthology.orchestra.run.smithi028:workunit test osd/osd-copy-from.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2021-05-12T09:26:09.929 INFO:tasks.workunit.client.0.smithi028.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T09:26:09.936 INFO:tasks.workunit.client.0.smithi028.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T09:26:09.937 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:26:09.937 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:26:09.937 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T09:26:09.937 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T09:26:09.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T09:26:09.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T09:26:09.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T09:26:09.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T09:26:09.939 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-copy-from 2021-05-12T09:26:09.939 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:23: run: local dir=td/osd-copy-from 2021-05-12T09:26:09.939 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:24: run: shift 2021-05-12T09:26:09.939 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: export CEPH_MON=127.0.0.1:7111 2021-05-12T09:26:09.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: CEPH_MON=127.0.0.1:7111 2021-05-12T09:26:09.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:27: run: export CEPH_ARGS 2021-05-12T09:26:09.940 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: uuidgen 2021-05-12T09:26:09.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: CEPH_ARGS+='--fsid=1c1c537f-eb90-43c1-ae42-c7a1ae5f22f8 --auth-supported=none ' 2021-05-12T09:26:09.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7111 ' 2021-05-12T09:26:09.941 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: set 2021-05-12T09:26:09.941 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T09:26:09.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: local funcs=TEST_copy_from 2021-05-12T09:26:09.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:32: run: for func in $funcs 2021-05-12T09:26:09.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:33: run: setup td/osd-copy-from 2021-05-12T09:26:09.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-copy-from 2021-05-12T09:26:09.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-copy-from 2021-05-12T09:26:09.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-05-12T09:26:09.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:26:09.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-05-12T09:26:09.946 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:26:09.952 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:26:09.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:26:09.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:26:09.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:26:09.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:26:09.953 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:26:09.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:26:09.954 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:26:09.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:26:09.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:26:09.954 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:26:09.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:09.955 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:26:09.955 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:26:09.955 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:09.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:26:09.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:26:09.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-05-12T09:26:09.959 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:26:09.959 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:09.959 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144662 2021-05-12T09:26:09.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.144662 2021-05-12T09:26:09.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:26:09.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:26:09.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-copy-from 2021-05-12T09:26:09.961 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:26:09.961 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:09.961 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144662 2021-05-12T09:26:09.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.144662 2021-05-12T09:26:09.961 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:26:09.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T09:26:09.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T09:26:09.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:26:09.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-copy-from 1' TERM HUP INT 2021-05-12T09:26:09.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:34: run: TEST_copy_from td/osd-copy-from 2021-05-12T09:26:09.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:40: TEST_copy_from: local dir=td/osd-copy-from 2021-05-12T09:26:09.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:42: TEST_copy_from: run_mon td/osd-copy-from a 2021-05-12T09:26:09.964 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-copy-from 2021-05-12T09:26:09.964 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:26:09.964 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:26:09.964 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:26:09.965 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-copy-from/a 2021-05-12T09:26:09.965 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-copy-from/a --run-dir=td/osd-copy-from 2021-05-12T09:26:10.319 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:26:10.320 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:26:10.320 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:26:10.320 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:26:10.320 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:10.321 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144662 2021-05-12T09:26:10.321 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.144662/$cluster-$name.asok' 2021-05-12T09:26:10.321 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-copy-from/a '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.144662/$cluster-$name.asok' --mon-cluster-log-file=td/osd-copy-from/log --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:26:10.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:26:10.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:26:10.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:26:10.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:26:10.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:26:10.565 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:26:10.565 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:26:10.565 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:26:10.565 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:26:10.565 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:26:10.566 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:10.566 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144662 2021-05-12T09:26:10.566 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.144662/ceph-mon.a.asok 2021-05-12T09:26:10.566 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:26:10.567 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.144662/ceph-mon.a.asok config get fsid 2021-05-12T09:26:10.672 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:26:10.672 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:26:10.672 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:26:10.673 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:26:10.673 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:26:10.673 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:26:10.673 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:26:10.673 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:26:10.674 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:26:10.674 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:10.674 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144662 2021-05-12T09:26:10.674 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.144662/ceph-mon.a.asok 2021-05-12T09:26:10.675 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:26:10.675 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.144662/ceph-mon.a.asok config get mon_host 2021-05-12T09:26:10.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:43: TEST_copy_from: run_mgr td/osd-copy-from x 2021-05-12T09:26:10.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-copy-from 2021-05-12T09:26:10.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:26:10.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:26:10.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:26:10.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-copy-from/x 2021-05-12T09:26:10.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:26:11.651 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:26:11.652 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:26:11.652 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:26:11.652 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:26:11.652 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:11.653 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144662 2021-05-12T09:26:11.653 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.144662/$cluster-$name.asok' 2021-05-12T09:26:11.653 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:26:11.654 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-copy-from/x '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.144662/$cluster-$name.asok' --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:26:11.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:44: TEST_copy_from: run_osd td/osd-copy-from 0 2021-05-12T09:26:11.678 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2021-05-12T09:26:11.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:26:11.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:26:11.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:26:11.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/0 2021-05-12T09:26:11.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1c1c537f-eb90-43c1-ae42-c7a1ae5f22f8 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2021-05-12T09:26:11.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:26:11.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:26:11.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:26:11.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/0' 2021-05-12T09:26:11.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/0/journal' 2021-05-12T09:26:11.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:26:11.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:26:11.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2021-05-12T09:26:11.683 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:26:11.683 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:26:11.683 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:26:11.683 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:26:11.684 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:11.684 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144662 2021-05-12T09:26:11.684 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.144662/$cluster-$name.asok' 2021-05-12T09:26:11.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.144662/$cluster-$name.asok' 2021-05-12T09:26:11.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:26:11.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:26:11.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:26:11.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2021-05-12T09:26:11.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2021-05-12T09:26:11.686 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:26:11.686 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:26:11.686 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:26:11.687 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:26:11.687 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:26:11.687 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-copy-from/0 2021-05-12T09:26:11.687 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:26:11.687 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=ddab81a8-9930-4502-a5d7-40f3e30b36ad 2021-05-12T09:26:11.688 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 ddab81a8-9930-4502-a5d7-40f3e30b36ad' 2021-05-12T09:26:11.688 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 ddab81a8-9930-4502-a5d7-40f3e30b36ad 2021-05-12T09:26:11.689 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:26:11.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAzn5tgedMFKRAA1QpJVaHOO35RjVf8PMBCDQ== 2021-05-12T09:26:11.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAzn5tgedMFKRAA1QpJVaHOO35RjVf8PMBCDQ=="}' 2021-05-12T09:26:11.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new ddab81a8-9930-4502-a5d7-40f3e30b36ad -i td/osd-copy-from/0/new.json 2021-05-12T09:26:12.012 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:26:12.023 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-copy-from/0/new.json 2021-05-12T09:26:12.023 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=1c1c537f-eb90-43c1-ae42-c7a1ae5f22f8 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.144662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAzn5tgedMFKRAA1QpJVaHOO35RjVf8PMBCDQ== --osd-uuid ddab81a8-9930-4502-a5d7-40f3e30b36ad 2021-05-12T09:26:12.046 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:12.045+0000 7faf3c31c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:12.048 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:12.047+0000 7faf3c31c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:12.049 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:12.048+0000 7faf3c31c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:12.050 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:12.049+0000 7faf3c31c0c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-05-12T09:26:12.051 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:12.049+0000 7faf3c31c0c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-05-12T09:26:12.051 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:12.049+0000 7faf3c31c0c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-05-12T09:26:12.084 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:12.083+0000 7faf3c31c0c0 -1 bluestore(td/osd-copy-from/0) _read_fsid unparsable uuid 2021-05-12T09:26:16.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-copy-from/0/keyring 2021-05-12T09:26:16.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:26:16.677 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:26:16.678 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:26:16.678 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-copy-from/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:26:17.182 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:26:17.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:26:17.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=1c1c537f-eb90-43c1-ae42-c7a1ae5f22f8 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.144662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:26:17.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:26:17.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:26:17.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:26:17.201 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:17.199+0000 7f0c37c1d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:17.203 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:17.201+0000 7f0c37c1d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:17.204 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:17.202+0000 7f0c37c1d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:17.559 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:26:17.559 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:26:17.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:26:17.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:26:17.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:26:17.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:17.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:26:17.561 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:26:17.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:17.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:26:17.721 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:17.720+0000 7f0c37c1d0c0 -1 Falling back to public interface 2021-05-12T09:26:17.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:26:18.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:26:18.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:18.964 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:26:18.965 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:26:18.965 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:18.965 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:26:19.351 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:26:19.928 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:19.926+0000 7f0c37c1d0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:26:20.352 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:26:20.353 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:20.353 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:26:20.353 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:26:20.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:20.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:26:20.755 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/145782,v1:127.0.0.1:6803/145782] [v2:127.0.0.1:6804/145782,v1:127.0.0.1:6805/145782] exists,up ddab81a8-9930-4502-a5d7-40f3e30b36ad 2021-05-12T09:26:20.760 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:26:20.760 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:26:20.760 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:26:20.760 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:45: TEST_copy_from: run_osd td/osd-copy-from 1 2021-05-12T09:26:20.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2021-05-12T09:26:20.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:26:20.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:26:20.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:26:20.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/1 2021-05-12T09:26:20.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1c1c537f-eb90-43c1-ae42-c7a1ae5f22f8 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2021-05-12T09:26:20.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:26:20.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:26:20.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:26:20.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/1' 2021-05-12T09:26:20.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/1/journal' 2021-05-12T09:26:20.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:26:20.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:26:20.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2021-05-12T09:26:20.764 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:26:20.764 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:26:20.764 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:26:20.765 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:26:20.765 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:20.765 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144662 2021-05-12T09:26:20.765 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.144662/$cluster-$name.asok' 2021-05-12T09:26:20.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.144662/$cluster-$name.asok' 2021-05-12T09:26:20.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:26:20.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:26:20.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:26:20.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2021-05-12T09:26:20.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2021-05-12T09:26:20.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:26:20.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:26:20.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:26:20.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:26:20.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:26:20.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-copy-from/1 2021-05-12T09:26:20.768 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:26:20.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=128e412f-2a16-4bd2-8ea7-8296571f92e9 2021-05-12T09:26:20.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 128e412f-2a16-4bd2-8ea7-8296571f92e9' 2021-05-12T09:26:20.769 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 128e412f-2a16-4bd2-8ea7-8296571f92e9 2021-05-12T09:26:20.769 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:26:20.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQA8n5tg7KHhLRAA1cTa/QlZykVe6zUV25b7vg== 2021-05-12T09:26:20.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQA8n5tg7KHhLRAA1cTa/QlZykVe6zUV25b7vg=="}' 2021-05-12T09:26:20.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 128e412f-2a16-4bd2-8ea7-8296571f92e9 -i td/osd-copy-from/1/new.json 2021-05-12T09:26:21.296 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:26:21.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-copy-from/1/new.json 2021-05-12T09:26:21.309 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=1c1c537f-eb90-43c1-ae42-c7a1ae5f22f8 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.144662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA8n5tg7KHhLRAA1cTa/QlZykVe6zUV25b7vg== --osd-uuid 128e412f-2a16-4bd2-8ea7-8296571f92e9 2021-05-12T09:26:21.331 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:21.329+0000 7fa718e5e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:21.355 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:21.353+0000 7fa718e5e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:21.356 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:21.354+0000 7fa718e5e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:21.357 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:21.355+0000 7fa718e5e0c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-05-12T09:26:21.357 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:21.355+0000 7fa718e5e0c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-05-12T09:26:21.357 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:21.355+0000 7fa718e5e0c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-05-12T09:26:21.422 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:21.420+0000 7fa718e5e0c0 -1 bluestore(td/osd-copy-from/1) _read_fsid unparsable uuid 2021-05-12T09:26:26.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-copy-from/1/keyring 2021-05-12T09:26:26.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:26:26.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:26:26.338 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:26:26.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-copy-from/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:26:26.853 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:26:26.854 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:26:26.854 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=1c1c537f-eb90-43c1-ae42-c7a1ae5f22f8 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.144662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:26:26.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:26:26.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:26:26.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:26:26.872 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:26.871+0000 7fca286680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:26.874 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:26.873+0000 7fca286680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:26.876 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:26.874+0000 7fca286680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:27.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:26:27.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:26:27.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:26:27.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:26:27.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:26:27.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:27.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:26:27.236 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:26:27.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:27.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:26:27.393 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:27.391+0000 7fca286680c0 -1 Falling back to public interface 2021-05-12T09:26:27.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:26:28.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:26:28.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:28.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:26:28.638 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:26:28.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:28.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:26:29.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:26:29.623 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:29.620+0000 7fca286680c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:26:30.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:26:30.029 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:30.029 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:26:30.029 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:26:30.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:30.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:26:30.444 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/147167,v1:127.0.0.1:6811/147167] [v2:127.0.0.1:6812/147167,v1:127.0.0.1:6813/147167] exists,up 128e412f-2a16-4bd2-8ea7-8296571f92e9 2021-05-12T09:26:30.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:26:30.445 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:26:30.445 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:26:30.445 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:46: TEST_copy_from: create_rbd_pool 2021-05-12T09:26:30.445 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T09:26:30.944 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'rbd' does not exist 2021-05-12T09:26:30.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T09:26:30.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T09:26:31.460 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'rbd' already exists 2021-05-12T09:26:31.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:26:32.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T09:26:33.497 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: which rados 2021-05-12T09:26:33.505 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: rados -p rbd put foo /usr/bin/rados 2021-05-12T09:26:33.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:50: TEST_copy_from: rados -p rbd cp foo foo2 2021-05-12T09:26:33.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:51: TEST_copy_from: rados -p rbd stat foo2 2021-05-12T09:26:33.831 INFO:tasks.workunit.client.0.smithi028.stdout:rbd/foo2 mtime 2021-05-12T09:26:33.000000+0000, size 825488 2021-05-12T09:26:33.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:54: TEST_copy_from: ceph tell 'osd.*' injectargs -- --osd-debug-inject-copyfrom-error 2021-05-12T09:26:34.021 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0: {} 2021-05-12T09:26:34.051 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1: {} 2021-05-12T09:26:34.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:55: TEST_copy_from: rados -p rbd cp foo foo3 2021-05-12T09:26:34.085 INFO:tasks.workunit.client.0.smithi028.stderr:error copying rbd/foo => rbd/foo3: (5) Input/output error 2021-05-12T09:26:34.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:56: TEST_copy_from: rados -p rbd stat foo3 2021-05-12T09:26:34.109 INFO:tasks.workunit.client.0.smithi028.stderr: error stat-ing rbd/foo3: (2) No such file or directory 2021-05-12T09:26:34.111 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:59: TEST_copy_from: ceph tell 'osd.*' injectargs -- --no-osd-debug-inject-copyfrom-error 2021-05-12T09:26:34.248 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0: {} 2021-05-12T09:26:34.277 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1: {} 2021-05-12T09:26:34.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:60: TEST_copy_from: rados -p rbd cp foo foo3 2021-05-12T09:26:34.413 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:61: TEST_copy_from: rados -p rbd stat foo3 2021-05-12T09:26:34.436 INFO:tasks.workunit.client.0.smithi028.stdout:rbd/foo3 mtime 2021-05-12T09:26:33.000000+0000, size 825488 2021-05-12T09:26:34.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:35: run: teardown td/osd-copy-from 2021-05-12T09:26:34.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-05-12T09:26:34.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:26:34.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-05-12T09:26:34.439 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:26:34.439 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:26:34.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:26:34.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:26:34.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:26:34.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:26:34.547 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:26:34.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:26:34.548 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:26:34.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:26:34.549 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:26:34.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:26:34.550 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:34.550 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:26:34.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:26:34.551 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:34.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:26:34.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:26:34.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-05-12T09:26:34.556 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:26:34.557 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:34.557 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144662 2021-05-12T09:26:34.557 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.144662 2021-05-12T09:26:34.559 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:26:34.559 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:26:34.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T09:26:34.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-copy-from 0 2021-05-12T09:26:34.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-05-12T09:26:34.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T09:26:34.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-05-12T09:26:34.561 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:26:34.561 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:26:34.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:26:34.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:26:34.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:26:34.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:26:34.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:26:34.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:26:34.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:26:34.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:26:34.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:26:34.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:26:34.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:34.566 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:26:34.566 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:26:34.566 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:34.566 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:26:34.567 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T09:26:34.567 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-05-12T09:26:34.569 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:26:34.569 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:34.569 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.144662 2021-05-12T09:26:34.569 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.144662 2021-05-12T09:26:34.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:26:34.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:26:34.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T09:26:34.570 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T09:26:34.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T09:26:34.634 INFO:tasks.workunit:Running workunit osd/osd-dup.sh... 2021-05-12T09:26:34.634 DEBUG:teuthology.orchestra.run.smithi028:workunit test osd/osd-dup.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2021-05-12T09:26:34.694 INFO:tasks.workunit.client.0.smithi028.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T09:26:34.703 INFO:tasks.workunit.client.0.smithi028.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T09:26:34.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:26:34.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:26:34.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T09:26:34.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T09:26:34.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T09:26:34.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T09:26:34.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T09:26:34.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T09:26:34.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-dup 2021-05-12T09:26:34.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:8: run: local dir=td/osd-dup 2021-05-12T09:26:34.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:9: run: shift 2021-05-12T09:26:34.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: export CEPH_MON=127.0.0.1:7146 2021-05-12T09:26:34.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: CEPH_MON=127.0.0.1:7146 2021-05-12T09:26:34.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:12: run: export CEPH_ARGS 2021-05-12T09:26:34.709 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: uuidgen 2021-05-12T09:26:34.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: CEPH_ARGS+='--fsid=f58b0e48-60a2-4bfc-9885-fcb9a5ed57df --auth-supported=none ' 2021-05-12T09:26:34.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2021-05-12T09:26:34.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:16: run: CEPH_ARGS+='--filestore_wbthrottle_xfs_ios_hard_limit=900 ' 2021-05-12T09:26:34.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:17: run: CEPH_ARGS+='--filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T09:26:34.710 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: set 2021-05-12T09:26:34.710 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T09:26:34.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: local funcs=TEST_filestore_to_bluestore 2021-05-12T09:26:34.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:19: run: for func in $funcs 2021-05-12T09:26:34.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:20: run: setup td/osd-dup 2021-05-12T09:26:34.715 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-dup 2021-05-12T09:26:34.715 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-dup 2021-05-12T09:26:34.715 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-05-12T09:26:34.715 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:26:34.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-05-12T09:26:34.716 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:26:34.716 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:26:34.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:26:34.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:26:34.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:26:34.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:26:34.717 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:26:34.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:26:34.717 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:26:34.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:26:34.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:26:34.718 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:26:34.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:34.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:26:34.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:26:34.719 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:26:34.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:26:34.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:26:34.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-05-12T09:26:34.720 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:26:34.720 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:34.720 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148052 2021-05-12T09:26:34.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.148052 2021-05-12T09:26:34.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:26:34.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:26:34.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-dup 2021-05-12T09:26:34.724 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:26:34.724 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:34.724 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148052 2021-05-12T09:26:34.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.148052 2021-05-12T09:26:34.725 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:26:34.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T09:26:34.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T09:26:34.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:26:34.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-dup 1' TERM HUP INT 2021-05-12T09:26:34.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:21: run: TEST_filestore_to_bluestore td/osd-dup 2021-05-12T09:26:34.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:27: TEST_filestore_to_bluestore: local dir=td/osd-dup 2021-05-12T09:26:34.726 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: ulimit -n 2021-05-12T09:26:34.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: local flimit=4096 2021-05-12T09:26:34.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:30: TEST_filestore_to_bluestore: '[' 4096 -lt 1536 ']' 2021-05-12T09:26:34.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:34: TEST_filestore_to_bluestore: run_mon td/osd-dup a 2021-05-12T09:26:34.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-dup 2021-05-12T09:26:34.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:26:34.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:26:34.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:26:34.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-dup/a 2021-05-12T09:26:34.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-dup/a --run-dir=td/osd-dup 2021-05-12T09:26:35.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:26:35.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:26:35.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:26:35.111 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:26:35.112 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:35.112 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148052 2021-05-12T09:26:35.112 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.148052/$cluster-$name.asok' 2021-05-12T09:26:35.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-dup/a '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.148052/$cluster-$name.asok' --mon-cluster-log-file=td/osd-dup/log --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:26:35.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:26:35.383 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:26:35.383 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:26:35.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:26:35.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:26:35.385 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:26:35.385 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:26:35.386 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:26:35.386 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:26:35.386 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:26:35.386 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:35.387 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148052 2021-05-12T09:26:35.387 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.148052/ceph-mon.a.asok 2021-05-12T09:26:35.387 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:26:35.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.148052/ceph-mon.a.asok config get fsid 2021-05-12T09:26:35.486 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:26:35.487 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:26:35.487 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:26:35.488 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:26:35.488 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:26:35.488 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:26:35.488 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:26:35.488 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:26:35.489 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:26:35.489 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:35.489 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148052 2021-05-12T09:26:35.489 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.148052/ceph-mon.a.asok 2021-05-12T09:26:35.490 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:26:35.490 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.148052/ceph-mon.a.asok config get mon_host 2021-05-12T09:26:35.586 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:35: TEST_filestore_to_bluestore: run_mgr td/osd-dup x 2021-05-12T09:26:35.586 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-dup 2021-05-12T09:26:35.587 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:26:35.587 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:26:35.588 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:26:35.588 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-dup/x 2021-05-12T09:26:35.588 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:26:36.469 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:26:36.470 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:26:36.470 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:26:36.471 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:26:36.471 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:36.471 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148052 2021-05-12T09:26:36.471 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.148052/$cluster-$name.asok' 2021-05-12T09:26:36.472 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:26:36.472 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-dup/x '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.148052/$cluster-$name.asok' --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:26:36.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:36: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 0 2021-05-12T09:26:36.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: local dir=td/osd-dup 2021-05-12T09:26:36.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: shift 2021-05-12T09:26:36.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: local id=0 2021-05-12T09:26:36.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: shift 2021-05-12T09:26:36.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: local osd_data=td/osd-dup/0 2021-05-12T09:26:36.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: local 'ceph_args=--fsid=f58b0e48-60a2-4bfc-9885-fcb9a5ed57df --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T09:26:36.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:26:36.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:26:36.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:26:36.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/0' 2021-05-12T09:26:36.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2021-05-12T09:26:36.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --chdir=' 2021-05-12T09:26:36.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+= 2021-05-12T09:26:36.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-05-12T09:26:36.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: get_asok_path 2021-05-12T09:26:36.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:26:36.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:26:36.500 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:26:36.500 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:36.501 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148052 2021-05-12T09:26:36.501 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.148052/$cluster-$name.asok' 2021-05-12T09:26:36.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.148052/$cluster-$name.asok' 2021-05-12T09:26:36.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-05-12T09:26:36.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-05-12T09:26:36.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-05-12T09:26:36.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-05-12T09:26:36.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-05-12T09:26:36.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:26:36.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:26:36.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:26:36.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' ' 2021-05-12T09:26:36.504 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+= 2021-05-12T09:26:36.504 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: mkdir -p td/osd-dup/0 2021-05-12T09:26:36.504 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: uuidgen 2021-05-12T09:26:36.504 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: local uuid=c4eb54fd-7afb-45a0-9cb8-a30e93fa32ef 2021-05-12T09:26:36.504 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: echo 'add osd c4eb54fd-7afb-45a0-9cb8-a30e93fa32ef' 2021-05-12T09:26:36.505 INFO:tasks.workunit.client.0.smithi028.stdout:add osd c4eb54fd-7afb-45a0-9cb8-a30e93fa32ef 2021-05-12T09:26:36.505 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: ceph-authtool --gen-print-key 2021-05-12T09:26:36.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: OSD_SECRET=AQBMn5tg8L1GHhAAGvbLncDBHq52NBfFjXPT0Q== 2021-05-12T09:26:36.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo '{"cephx_secret": "AQBMn5tg8L1GHhAAGvbLncDBHq52NBfFjXPT0Q=="}' 2021-05-12T09:26:36.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph osd new c4eb54fd-7afb-45a0-9cb8-a30e93fa32ef -i td/osd-dup/0/new.json 2021-05-12T09:26:36.828 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:26:36.839 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: rm td/osd-dup/0/new.json 2021-05-12T09:26:36.842 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph-osd -i 0 --fsid=f58b0e48-60a2-4bfc-9885-fcb9a5ed57df --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.148052/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBMn5tg8L1GHhAAGvbLncDBHq52NBfFjXPT0Q== --osd-uuid c4eb54fd-7afb-45a0-9cb8-a30e93fa32ef --osd-objectstore=filestore 2021-05-12T09:26:36.865 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:36.861+0000 7f44255fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:36.866 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:36.863+0000 7f44255fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:36.869 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:36.864+0000 7f44255fc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:37.272 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:37.268+0000 7f44255fc0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T09:26:37.749 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:37.746+0000 7f44255fc0c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T09:26:37.750 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:37.747+0000 7f44255fc0c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T09:26:37.750 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:37.747+0000 7f44255fc0c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T09:26:38.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: local key_fn=td/osd-dup/0/keyring 2021-05-12T09:26:38.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: cat 2021-05-12T09:26:38.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: echo adding osd0 key to auth repository 2021-05-12T09:26:38.164 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:26:38.164 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: ceph -i td/osd-dup/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:26:38.338 INFO:tasks.workunit.client.0.smithi028.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2021-05-12T09:26:38.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo start osd.0 2021-05-12T09:26:38.341 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:26:38.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph-osd -i 0 --fsid=f58b0e48-60a2-4bfc-9885-fcb9a5ed57df --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.148052/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:26:38.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: ceph osd dump --format=json 2021-05-12T09:26:38.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: jq '.flags_set[]' 2021-05-12T09:26:38.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: grep -q '"noup"' 2021-05-12T09:26:38.362 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:38.360+0000 7f814b5d70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:38.364 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:38.362+0000 7f814b5d70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:38.366 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:38.364+0000 7f814b5d70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:38.367 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:38.365+0000 7f814b5d70c0 -1 Falling back to public interface 2021-05-12T09:26:38.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:738: run_osd_filestore: wait_for_osd up 0 2021-05-12T09:26:38.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:26:38.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:26:38.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:26:38.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:26:38.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:38.659 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:26:38.660 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:26:38.660 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:38.660 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:26:38.796 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:38.793+0000 7f814b5d70c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T09:26:38.796 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:38.793+0000 7f814b5d70c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T09:26:38.797 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:38.793+0000 7f814b5d70c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T09:26:38.802 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:38.800+0000 7f814b5d70c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:26:38.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:26:39.891 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:26:39.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:39.892 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:26:39.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:26:39.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:39.894 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:26:40.141 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/148375,v1:127.0.0.1:6801/148375] [v2:127.0.0.1:6802/148375,v1:127.0.0.1:6803/148375] exists,up c4eb54fd-7afb-45a0-9cb8-a30e93fa32ef 2021-05-12T09:26:40.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:26:40.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:26:40.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:26:40.149 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: cat td/osd-dup/osd.0.pid 2021-05-12T09:26:40.149 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: osd_pid=148375 2021-05-12T09:26:40.149 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:38: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 1 2021-05-12T09:26:40.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: local dir=td/osd-dup 2021-05-12T09:26:40.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: shift 2021-05-12T09:26:40.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: local id=1 2021-05-12T09:26:40.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: shift 2021-05-12T09:26:40.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: local osd_data=td/osd-dup/1 2021-05-12T09:26:40.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: local 'ceph_args=--fsid=f58b0e48-60a2-4bfc-9885-fcb9a5ed57df --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T09:26:40.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:26:40.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:26:40.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:26:40.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/1' 2021-05-12T09:26:40.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/1/journal' 2021-05-12T09:26:40.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --chdir=' 2021-05-12T09:26:40.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+= 2021-05-12T09:26:40.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-05-12T09:26:40.154 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: get_asok_path 2021-05-12T09:26:40.154 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:26:40.154 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:26:40.155 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:26:40.155 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:40.155 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148052 2021-05-12T09:26:40.155 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.148052/$cluster-$name.asok' 2021-05-12T09:26:40.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.148052/$cluster-$name.asok' 2021-05-12T09:26:40.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-05-12T09:26:40.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-05-12T09:26:40.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-05-12T09:26:40.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-05-12T09:26:40.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-05-12T09:26:40.158 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:26:40.158 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:26:40.158 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:26:40.159 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' ' 2021-05-12T09:26:40.159 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+= 2021-05-12T09:26:40.159 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: mkdir -p td/osd-dup/1 2021-05-12T09:26:40.160 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: uuidgen 2021-05-12T09:26:40.160 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: local uuid=5025fc99-97b2-4449-9005-dff7aee4af9e 2021-05-12T09:26:40.160 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: echo 'add osd 5025fc99-97b2-4449-9005-dff7aee4af9e' 2021-05-12T09:26:40.161 INFO:tasks.workunit.client.0.smithi028.stdout:add osd 5025fc99-97b2-4449-9005-dff7aee4af9e 2021-05-12T09:26:40.162 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: ceph-authtool --gen-print-key 2021-05-12T09:26:40.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: OSD_SECRET=AQBQn5tg7+NoCRAAEcY93ucs007MiFXsgRBhcg== 2021-05-12T09:26:40.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo '{"cephx_secret": "AQBQn5tg7+NoCRAAEcY93ucs007MiFXsgRBhcg=="}' 2021-05-12T09:26:40.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph osd new 5025fc99-97b2-4449-9005-dff7aee4af9e -i td/osd-dup/1/new.json 2021-05-12T09:26:40.524 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:26:40.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: rm td/osd-dup/1/new.json 2021-05-12T09:26:40.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph-osd -i 1 --fsid=f58b0e48-60a2-4bfc-9885-fcb9a5ed57df --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.148052/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBQn5tg7+NoCRAAEcY93ucs007MiFXsgRBhcg== --osd-uuid 5025fc99-97b2-4449-9005-dff7aee4af9e --osd-objectstore=filestore 2021-05-12T09:26:40.561 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:40.558+0000 7f2e35ec20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:40.562 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:40.559+0000 7f2e35ec20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:40.564 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:40.561+0000 7f2e35ec20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:40.993 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:40.990+0000 7f2e35ec20c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T09:26:41.563 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:41.560+0000 7f2e35ec20c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T09:26:41.571 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:41.569+0000 7f2e35ec20c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T09:26:41.572 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:41.569+0000 7f2e35ec20c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T09:26:42.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: local key_fn=td/osd-dup/1/keyring 2021-05-12T09:26:42.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: cat 2021-05-12T09:26:42.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: echo adding osd1 key to auth repository 2021-05-12T09:26:42.947 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:26:42.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: ceph -i td/osd-dup/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:26:42.948 INFO:tasks.workunit.client.0.smithi028.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2021-05-12T09:26:42.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo start osd.1 2021-05-12T09:26:42.949 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:26:42.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph-osd -i 1 --fsid=f58b0e48-60a2-4bfc-9885-fcb9a5ed57df --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.148052/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:26:42.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: ceph osd dump --format=json 2021-05-12T09:26:42.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: jq '.flags_set[]' 2021-05-12T09:26:42.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: grep -q '"noup"' 2021-05-12T09:26:42.952 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:42.529+0000 7f5d6ea450c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:42.952 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:42.531+0000 7f5d6ea450c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:42.952 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:42.532+0000 7f5d6ea450c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:42.952 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:42.534+0000 7f5d6ea450c0 -1 Falling back to public interface 2021-05-12T09:26:42.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:738: run_osd_filestore: wait_for_osd up 1 2021-05-12T09:26:42.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:26:42.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:26:42.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:26:42.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:26:42.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:42.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:26:42.955 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:26:42.956 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:42.957 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:26:42.957 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:42.941+0000 7f5d6ea450c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T09:26:42.958 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:42.941+0000 7f5d6ea450c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T09:26:42.958 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:42.941+0000 7f5d6ea450c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T09:26:42.958 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:42.947+0000 7f5d6ea450c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:26:43.306 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:26:44.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:26:44.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:44.309 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:26:44.310 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:26:44.310 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:44.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:26:44.696 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/148755,v1:127.0.0.1:6811/148755] [v2:127.0.0.1:6812/148755,v1:127.0.0.1:6813/148755] exists,up 5025fc99-97b2-4449-9005-dff7aee4af9e 2021-05-12T09:26:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:26:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:26:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:26:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:39: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 2 2021-05-12T09:26:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: local dir=td/osd-dup 2021-05-12T09:26:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: shift 2021-05-12T09:26:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: local id=2 2021-05-12T09:26:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: shift 2021-05-12T09:26:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: local osd_data=td/osd-dup/2 2021-05-12T09:26:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: local 'ceph_args=--fsid=f58b0e48-60a2-4bfc-9885-fcb9a5ed57df --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T09:26:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:26:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:26:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:26:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/2' 2021-05-12T09:26:44.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/2/journal' 2021-05-12T09:26:44.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --chdir=' 2021-05-12T09:26:44.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+= 2021-05-12T09:26:44.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-05-12T09:26:44.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: get_asok_path 2021-05-12T09:26:44.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:26:44.707 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:26:44.707 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:26:44.707 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:26:44.707 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148052 2021-05-12T09:26:44.707 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.148052/$cluster-$name.asok' 2021-05-12T09:26:44.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.148052/$cluster-$name.asok' 2021-05-12T09:26:44.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-05-12T09:26:44.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-05-12T09:26:44.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-05-12T09:26:44.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-05-12T09:26:44.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-05-12T09:26:44.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:26:44.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:26:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:26:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' ' 2021-05-12T09:26:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+= 2021-05-12T09:26:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: mkdir -p td/osd-dup/2 2021-05-12T09:26:44.711 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: uuidgen 2021-05-12T09:26:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: local uuid=d8e3623b-f4bf-42d6-856a-06ebd008de40 2021-05-12T09:26:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: echo 'add osd d8e3623b-f4bf-42d6-856a-06ebd008de40' 2021-05-12T09:26:44.712 INFO:tasks.workunit.client.0.smithi028.stdout:add osd d8e3623b-f4bf-42d6-856a-06ebd008de40 2021-05-12T09:26:44.712 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: ceph-authtool --gen-print-key 2021-05-12T09:26:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: OSD_SECRET=AQBUn5tg8jmRKhAAgjgGwn4OD/izA9AXPmrr7A== 2021-05-12T09:26:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo '{"cephx_secret": "AQBUn5tg8jmRKhAAgjgGwn4OD/izA9AXPmrr7A=="}' 2021-05-12T09:26:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph osd new d8e3623b-f4bf-42d6-856a-06ebd008de40 -i td/osd-dup/2/new.json 2021-05-12T09:26:45.193 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:26:45.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: rm td/osd-dup/2/new.json 2021-05-12T09:26:45.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph-osd -i 2 --fsid=f58b0e48-60a2-4bfc-9885-fcb9a5ed57df --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.148052/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBUn5tg8jmRKhAAgjgGwn4OD/izA9AXPmrr7A== --osd-uuid d8e3623b-f4bf-42d6-856a-06ebd008de40 --osd-objectstore=filestore 2021-05-12T09:26:45.229 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:45.226+0000 7fe38fa510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:45.231 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:45.228+0000 7fe38fa510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:45.232 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:45.230+0000 7fe38fa510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:45.793 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:45.791+0000 7fe38fa510c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T09:26:46.288 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:46.286+0000 7fe38fa510c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T09:26:46.289 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:46.286+0000 7fe38fa510c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T09:26:46.289 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:46.286+0000 7fe38fa510c0 -1 journal do_read_entry(4096): bad header magic 2021-05-12T09:26:46.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: local key_fn=td/osd-dup/2/keyring 2021-05-12T09:26:46.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: cat 2021-05-12T09:26:46.722 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd2 key to auth repository 2021-05-12T09:26:46.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: echo adding osd2 key to auth repository 2021-05-12T09:26:46.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: ceph -i td/osd-dup/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:26:47.058 INFO:tasks.workunit.client.0.smithi028.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2021-05-12T09:26:47.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo start osd.2 2021-05-12T09:26:47.060 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:26:47.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph-osd -i 2 --fsid=f58b0e48-60a2-4bfc-9885-fcb9a5ed57df --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.148052/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:26:47.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: ceph osd dump --format=json 2021-05-12T09:26:47.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: jq '.flags_set[]' 2021-05-12T09:26:47.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:735: run_osd_filestore: grep -q '"noup"' 2021-05-12T09:26:47.081 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:47.079+0000 7fb9480000c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:47.082 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:47.081+0000 7fb9480000c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:47.084 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:47.083+0000 7fb9480000c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:26:47.086 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:47.084+0000 7fb9480000c0 -1 Falling back to public interface 2021-05-12T09:26:47.551 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:47.549+0000 7fb9480000c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-05-12T09:26:47.559 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:47.557+0000 7fb9480000c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T09:26:47.559 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:47.557+0000 7fb9480000c0 -1 journal do_read_entry(8192): bad header magic 2021-05-12T09:26:47.565 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:26:47.563+0000 7fb9480000c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T09:26:47.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:738: run_osd_filestore: wait_for_osd up 2 2021-05-12T09:26:47.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:26:47.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:26:47.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:26:47.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:26:47.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:47.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:26:47.582 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:26:47.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:47.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:26:47.976 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:26:48.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:26:48.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:26:48.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:26:48.979 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:26:48.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:26:48.980 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:26:49.371 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/149047,v1:127.0.0.1:6819/149047] [v2:127.0.0.1:6820/149047,v1:127.0.0.1:6821/149047] exists,up d8e3623b-f4bf-42d6-856a-06ebd008de40 2021-05-12T09:26:49.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:26:49.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:26:49.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:26:49.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:41: TEST_filestore_to_bluestore: sleep 5 2021-05-12T09:26:54.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:43: TEST_filestore_to_bluestore: create_pool foo 16 2021-05-12T09:26:54.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2021-05-12T09:26:55.101 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'foo' already exists 2021-05-12T09:26:55.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:26:56.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:46: TEST_filestore_to_bluestore: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2021-05-12T09:26:56.144 INFO:tasks.workunit.client.0.smithi028.stdout:hints = 1 2021-05-12T09:26:56.145 INFO:tasks.workunit.client.0.smithi028.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2021-05-12T09:26:56.145 INFO:tasks.workunit.client.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_149239 2021-05-12T09:26:56.145 INFO:tasks.workunit.client.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T09:26:56.145 INFO:tasks.workunit.client.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2021-05-12T09:26:57.144 INFO:tasks.workunit.client.0.smithi028.stdout: 1 16 16 0 0 0 - 0 2021-05-12T09:26:58.145 INFO:tasks.workunit.client.0.smithi028.stdout: 2 16 16 0 0 0 - 0 2021-05-12T09:26:59.145 INFO:tasks.workunit.client.0.smithi028.stdout: 3 16 16 0 0 0 - 0 2021-05-12T09:27:00.145 INFO:tasks.workunit.client.0.smithi028.stdout: 4 16 16 0 0 0 - 0 2021-05-12T09:27:01.145 INFO:tasks.workunit.client.0.smithi028.stdout: 5 16 117 101 0.0788958 0.0789062 0.11119 0.784364 2021-05-12T09:27:02.145 INFO:tasks.workunit.client.0.smithi028.stdout: 6 16 231 215 0.139955 0.445312 0.1208 0.439024 2021-05-12T09:27:03.145 INFO:tasks.workunit.client.0.smithi028.stdout: 7 16 359 343 0.191381 0.5 0.128498 0.323583 2021-05-12T09:27:04.145 INFO:tasks.workunit.client.0.smithi028.stdout: 8 16 437 421 0.205539 0.304688 0.430443 0.302694 2021-05-12T09:27:05.146 INFO:tasks.workunit.client.0.smithi028.stdout: 9 16 519 503 0.218287 0.320312 0.178503 0.280895 2021-05-12T09:27:06.146 INFO:tasks.workunit.client.0.smithi028.stdout: 10 16 599 583 0.227706 0.3125 0.128434 0.27222 2021-05-12T09:27:07.146 INFO:tasks.workunit.client.0.smithi028.stdout:Total time run: 10.0835 2021-05-12T09:27:07.146 INFO:tasks.workunit.client.0.smithi028.stdout:Total writes made: 599 2021-05-12T09:27:07.146 INFO:tasks.workunit.client.0.smithi028.stdout:Write size: 4096 2021-05-12T09:27:07.147 INFO:tasks.workunit.client.0.smithi028.stdout:Object size: 4096 2021-05-12T09:27:07.147 INFO:tasks.workunit.client.0.smithi028.stdout:Bandwidth (MB/sec): 0.232046 2021-05-12T09:27:07.147 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev Bandwidth: 0.200772 2021-05-12T09:27:07.147 INFO:tasks.workunit.client.0.smithi028.stdout:Max bandwidth (MB/sec): 0.5 2021-05-12T09:27:07.148 INFO:tasks.workunit.client.0.smithi028.stdout:Min bandwidth (MB/sec): 0 2021-05-12T09:27:07.148 INFO:tasks.workunit.client.0.smithi028.stdout:Average IOPS: 59 2021-05-12T09:27:07.148 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev IOPS: 51.4108 2021-05-12T09:27:07.148 INFO:tasks.workunit.client.0.smithi028.stdout:Max IOPS: 128 2021-05-12T09:27:07.149 INFO:tasks.workunit.client.0.smithi028.stdout:Min IOPS: 0 2021-05-12T09:27:07.149 INFO:tasks.workunit.client.0.smithi028.stdout:Average Latency(s): 0.268117 2021-05-12T09:27:07.149 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev Latency(s): 0.665589 2021-05-12T09:27:07.149 INFO:tasks.workunit.client.0.smithi028.stdout:Max latency(s): 4.31676 2021-05-12T09:27:07.149 INFO:tasks.workunit.client.0.smithi028.stdout:Min latency(s): 0.11054 2021-05-12T09:27:07.206 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 148375 2021-05-12T09:27:07.207 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: sleep 1 2021-05-12T09:27:08.208 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 148375 2021-05-12T09:27:08.208 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh: line 49: kill: (148375) - No such process 2021-05-12T09:27:08.208 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:50: TEST_filestore_to_bluestore: ceph osd down 0 2021-05-12T09:27:08.606 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already down. 2021-05-12T09:27:08.615 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:52: TEST_filestore_to_bluestore: mv td/osd-dup/0 td/osd-dup/0.old 2021-05-12T09:27:08.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:53: TEST_filestore_to_bluestore: mkdir td/osd-dup/0 2021-05-12T09:27:08.618 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: cat td/osd-dup/0.old/fsid 2021-05-12T09:27:08.619 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: ofsid=c4eb54fd-7afb-45a0-9cb8-a30e93fa32ef 2021-05-12T09:27:08.619 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:55: TEST_filestore_to_bluestore: echo 'osd fsid c4eb54fd-7afb-45a0-9cb8-a30e93fa32ef' 2021-05-12T09:27:08.620 INFO:tasks.workunit.client.0.smithi028.stdout:osd fsid c4eb54fd-7afb-45a0-9cb8-a30e93fa32ef 2021-05-12T09:27:08.620 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:56: TEST_filestore_to_bluestore: O='--fsid=f58b0e48-60a2-4bfc-9885-fcb9a5ed57df --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T09:27:08.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:57: TEST_filestore_to_bluestore: CEPH_ARGS+='--log-file td/osd-dup/cot.log --log-max-recent 0 ' 2021-05-12T09:27:08.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:58: TEST_filestore_to_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-dup/0 --fsid c4eb54fd-7afb-45a0-9cb8-a30e93fa32ef --op mkfs --no-mon-config 2021-05-12T09:27:12.265 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:60: TEST_filestore_to_bluestore: ceph-objectstore-tool --data-path td/osd-dup/0.old --target-data-path td/osd-dup/0 --op dup 2021-05-12T09:27:12.290 INFO:tasks.workunit.client.0.smithi028.stdout:dup from filestore: td/osd-dup/0.old 2021-05-12T09:27:12.290 INFO:tasks.workunit.client.0.smithi028.stdout: to bluestore: td/osd-dup/0 2021-05-12T09:27:13.192 INFO:tasks.workunit.client.0.smithi028.stdout:fsid c4eb54fd-7afb-45a0-9cb8-a30e93fa32ef 2021-05-12T09:27:13.192 INFO:tasks.workunit.client.0.smithi028.stdout:18 collections 2021-05-12T09:27:13.192 INFO:tasks.workunit.client.0.smithi028.stdout:1/18 2.a_head 2021-05-12T09:27:13.558 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 33 objects, 131072 bytes, 38 keys 2021-05-12T09:27:13.558 INFO:tasks.workunit.client.0.smithi028.stdout:2/18 2.9_head 2021-05-12T09:27:13.835 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 34 objects, 135168 bytes, 39 keys 2021-05-12T09:27:13.836 INFO:tasks.workunit.client.0.smithi028.stdout:3/18 2.7_head 2021-05-12T09:27:14.104 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 38 objects, 151552 bytes, 43 keys 2021-05-12T09:27:14.104 INFO:tasks.workunit.client.0.smithi028.stdout:4/18 2.0_head 2021-05-12T09:27:14.349 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 37 objects, 147456 bytes, 42 keys 2021-05-12T09:27:14.349 INFO:tasks.workunit.client.0.smithi028.stdout:5/18 2.b_head 2021-05-12T09:27:14.353 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 44 objects, 176128 bytes, 49 keys 2021-05-12T09:27:14.353 INFO:tasks.workunit.client.0.smithi028.stdout:6/18 2.f_head 2021-05-12T09:27:14.357 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 38 objects, 151552 bytes, 43 keys 2021-05-12T09:27:14.357 INFO:tasks.workunit.client.0.smithi028.stdout:7/18 2.e_head 2021-05-12T09:27:14.526 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 36 objects, 143360 bytes, 41 keys 2021-05-12T09:27:14.527 INFO:tasks.workunit.client.0.smithi028.stdout:8/18 2.d_head 2021-05-12T09:27:14.769 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 40 objects, 155672 bytes, 45 keys 2021-05-12T09:27:14.770 INFO:tasks.workunit.client.0.smithi028.stdout:9/18 2.1_head 2021-05-12T09:27:14.987 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 39 objects, 155648 bytes, 44 keys 2021-05-12T09:27:14.988 INFO:tasks.workunit.client.0.smithi028.stdout:10/18 meta 2021-05-12T09:27:14.992 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 52 objects, 57904 bytes, 0 keys 2021-05-12T09:27:14.992 INFO:tasks.workunit.client.0.smithi028.stdout:11/18 2.3_head 2021-05-12T09:27:15.274 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 41 objects, 163840 bytes, 46 keys 2021-05-12T09:27:15.274 INFO:tasks.workunit.client.0.smithi028.stdout:12/18 2.6_head 2021-05-12T09:27:15.587 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 46 objects, 184320 bytes, 51 keys 2021-05-12T09:27:15.587 INFO:tasks.workunit.client.0.smithi028.stdout:13/18 2.8_head 2021-05-12T09:27:15.591 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 44 objects, 176128 bytes, 49 keys 2021-05-12T09:27:15.591 INFO:tasks.workunit.client.0.smithi028.stdout:14/18 2.5_head 2021-05-12T09:27:15.854 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 42 objects, 167936 bytes, 47 keys 2021-05-12T09:27:15.854 INFO:tasks.workunit.client.0.smithi028.stdout:15/18 2.2_head 2021-05-12T09:27:15.857 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 34 objects, 135168 bytes, 39 keys 2021-05-12T09:27:15.857 INFO:tasks.workunit.client.0.smithi028.stdout:16/18 2.c_head 2021-05-12T09:27:16.105 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 40 objects, 159744 bytes, 45 keys 2021-05-12T09:27:16.105 INFO:tasks.workunit.client.0.smithi028.stdout:17/18 2.4_head 2021-05-12T09:27:16.233 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 30 objects, 118784 bytes, 35 keys 2021-05-12T09:27:16.233 INFO:tasks.workunit.client.0.smithi028.stdout:18/18 1.0_head 2021-05-12T09:27:16.285 INFO:tasks.workunit.client.0.smithi028.stdout: 0 objects, 0 bytes, 0 keys 3 objects, 459280 bytes, 68 keys 2021-05-12T09:27:16.285 INFO:tasks.workunit.client.0.smithi028.stdout:keyring 2021-05-12T09:27:16.285 INFO:tasks.workunit.client.0.smithi028.stdout:duping osd metadata 2021-05-12T09:27:16.642 INFO:tasks.workunit.client.0.smithi028.stdout:done. 2021-05-12T09:27:17.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:62: TEST_filestore_to_bluestore: CEPH_ARGS='--fsid=f58b0e48-60a2-4bfc-9885-fcb9a5ed57df --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T09:27:17.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:64: TEST_filestore_to_bluestore: activate_osd td/osd-dup 0 2021-05-12T09:27:17.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-dup 2021-05-12T09:27:17.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:27:17.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T09:27:17.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:27:17.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-dup/0 2021-05-12T09:27:17.370 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=f58b0e48-60a2-4bfc-9885-fcb9a5ed57df --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-05-12T09:27:17.370 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:27:17.370 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:27:17.370 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:27:17.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-dup/0' 2021-05-12T09:27:17.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2021-05-12T09:27:17.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:27:17.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:27:17.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-dup' 2021-05-12T09:27:17.372 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:27:17.372 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:27:17.373 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:27:17.373 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:27:17.373 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:27:17.373 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148052 2021-05-12T09:27:17.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.148052/$cluster-$name.asok' 2021-05-12T09:27:17.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.148052/$cluster-$name.asok' 2021-05-12T09:27:17.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:27:17.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-05-12T09:27:17.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-05-12T09:27:17.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:27:17.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:27:17.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:27:17.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:27:17.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:27:17.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-dup/0 2021-05-12T09:27:17.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T09:27:17.378 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:27:17.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=f58b0e48-60a2-4bfc-9885-fcb9a5ed57df --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.148052/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:27:17.379 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-dup/0/whoami 2021-05-12T09:27:17.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T09:27:17.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:27:17.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:27:17.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:27:17.394 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:17.391+0000 7f2dad6080c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:27:17.402 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:17.399+0000 7f2dad6080c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:27:17.403 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:17.400+0000 7f2dad6080c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:27:17.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T09:27:17.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:27:17.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:27:17.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:27:17.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:27:17.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:27:17.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:27:17.764 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:27:17.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:27:17.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:27:17.928 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:17.926+0000 7f2dad6080c0 -1 Falling back to public interface 2021-05-12T09:27:18.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:27:19.141 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:27:19.142 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:27:19.142 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:27:19.143 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:27:19.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:27:19.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:27:19.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:27:20.534 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:27:20.534 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:27:20.534 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:27:20.535 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:27:20.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:27:20.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:27:20.538 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:20.535+0000 7f2dad6080c0 -1 osd.0 24 log_to_monitors {default=true} 2021-05-12T09:27:21.107 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 29 up_thru 21 down_at 25 last_clean_interval [5,24) [v2:127.0.0.1:6800/150122,v1:127.0.0.1:6801/150122] [v2:127.0.0.1:6802/150122,v1:127.0.0.1:6803/150122] exists,up c4eb54fd-7afb-45a0-9cb8-a30e93fa32ef 2021-05-12T09:27:21.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:27:21.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:27:21.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:27:21.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: ceph osd stat 2021-05-12T09:27:21.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: grep '3 up' 2021-05-12T09:27:21.728 INFO:tasks.workunit.client.0.smithi028.stdout:3 osds: 3 up (since 0.906237s), 3 in (since 36s); epoch: e31 2021-05-12T09:27:21.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: ceph osd metadata 0 2021-05-12T09:27:21.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: grep bluestore 2021-05-12T09:27:22.429 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_access_mode": "file", 2021-05-12T09:27:22.429 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_block_size": "4096", 2021-05-12T09:27:22.430 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_devices": "sda", 2021-05-12T09:27:22.430 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-05-12T09:27:22.430 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_path": "td/osd-dup/0/block", 2021-05-12T09:27:22.430 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_rotational": "1", 2021-05-12T09:27:22.431 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_size": "107374182400", 2021-05-12T09:27:22.431 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_support_discard": "0", 2021-05-12T09:27:22.431 INFO:tasks.workunit.client.0.smithi028.stdout: "bluestore_bdev_type": "hdd", 2021-05-12T09:27:22.431 INFO:tasks.workunit.client.0.smithi028.stdout: "osd_objectstore": "bluestore", 2021-05-12T09:27:22.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:69: TEST_filestore_to_bluestore: ceph osd scrub 0 2021-05-12T09:27:22.813 INFO:tasks.workunit.client.0.smithi028.stderr:instructed osd(s) 0 to scrub 2021-05-12T09:27:22.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:72: TEST_filestore_to_bluestore: sleep 15 2021-05-12T09:27:37.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:74: TEST_filestore_to_bluestore: flush_pg_stats 2021-05-12T09:27:37.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:27:37.824 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:27:38.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:27:38.219 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:27:38.220 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:27:38.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:27:38.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:27:38.221 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:27:38.364 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=124554051590 2021-05-12T09:27:38.687 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 124554051590 2021-05-12T09:27:38.687 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-124554051590' 2021-05-12T09:27:38.687 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:27:38.687 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:27:38.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672973 2021-05-12T09:27:38.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672973 2021-05-12T09:27:38.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-124554051590 1-42949672973' 2021-05-12T09:27:38.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:27:38.691 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:27:38.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509452 2021-05-12T09:27:38.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509452 2021-05-12T09:27:38.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-124554051590 1-42949672973 2-64424509452' 2021-05-12T09:27:38.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:27:38.692 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-124554051590 2021-05-12T09:27:38.692 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:27:38.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:27:38.693 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-124554051590 2021-05-12T09:27:38.693 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:27:38.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=124554051590 2021-05-12T09:27:38.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 124554051590' 2021-05-12T09:27:38.694 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 124554051590 2021-05-12T09:27:38.694 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:27:39.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 124554051588 -lt 124554051590 2021-05-12T09:27:39.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:27:40.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:27:40.007 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:27:40.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 124554051590 -lt 124554051590 2021-05-12T09:27:40.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:27:40.390 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672973 2021-05-12T09:27:40.390 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:27:40.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:27:40.392 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672973 2021-05-12T09:27:40.392 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:27:40.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672973 2021-05-12T09:27:40.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2021-05-12T09:27:40.394 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672973 2021-05-12T09:27:40.394 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:27:40.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672973 -lt 42949672973 2021-05-12T09:27:40.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:27:40.794 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2021-05-12T09:27:40.795 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:27:40.795 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:27:40.796 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509452 2021-05-12T09:27:40.796 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:27:40.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509452 2021-05-12T09:27:40.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-05-12T09:27:40.798 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509452 2021-05-12T09:27:40.799 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:27:41.172 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509452 -lt 64424509452 2021-05-12T09:27:41.172 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:76: TEST_filestore_to_bluestore: wait_for_clean 2021-05-12T09:27:41.172 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:27:41.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:27:41.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:27:41.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:27:41.174 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T09:27:41.175 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:27:41.175 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:27:41.175 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:27:41.175 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:27:41.176 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:27:41.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:27:41.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:27:41.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:27:41.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:27:41.264 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:27:41.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:27:41.644 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:27:41.644 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:27:41.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:27:41.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:27:41.645 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:27:41.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=124554051593 2021-05-12T09:27:41.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 124554051593 2021-05-12T09:27:41.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-124554051593' 2021-05-12T09:27:41.800 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:27:41.800 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:27:41.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672976 2021-05-12T09:27:41.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672976 2021-05-12T09:27:41.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-124554051593 1-42949672976' 2021-05-12T09:27:41.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:27:41.930 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:27:42.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509455 2021-05-12T09:27:42.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509455 2021-05-12T09:27:42.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-124554051593 1-42949672976 2-64424509455' 2021-05-12T09:27:42.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:27:42.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-124554051593 2021-05-12T09:27:42.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:27:42.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:27:42.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-124554051593 2021-05-12T09:27:42.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:27:42.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=124554051593 2021-05-12T09:27:42.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 124554051593' 2021-05-12T09:27:42.064 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 124554051593 2021-05-12T09:27:42.064 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:27:42.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 124554051591 -lt 124554051593 2021-05-12T09:27:42.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:27:43.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:27:43.439 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:27:43.850 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 124554051593 -lt 124554051593 2021-05-12T09:27:43.850 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:27:43.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672976 2021-05-12T09:27:43.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:27:43.853 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:27:43.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672976 2021-05-12T09:27:43.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:27:43.854 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672976 2021-05-12T09:27:43.854 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672976 2021-05-12T09:27:43.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2021-05-12T09:27:43.856 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:27:44.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672976 -lt 42949672976 2021-05-12T09:27:44.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:27:44.251 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509455 2021-05-12T09:27:44.251 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:27:44.252 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:27:44.253 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509455 2021-05-12T09:27:44.253 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:27:44.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509455 2021-05-12T09:27:44.254 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509455 2021-05-12T09:27:44.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2021-05-12T09:27:44.255 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:27:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509455 -lt 64424509455 2021-05-12T09:27:44.671 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:27:44.672 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:27:44.672 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:27:45.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 17 == 0 2021-05-12T09:27:45.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:27:45.194 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:27:45.195 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:27:45.195 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:27:45.195 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:27:45.195 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:27:45.196 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:27:45.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=17 2021-05-12T09:27:45.565 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:27:45.566 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:27:45.566 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:27:46.067 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 17 = 17 2021-05-12T09:27:46.067 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:27:46.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:27:46.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:22: run: teardown td/osd-dup 2021-05-12T09:27:46.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-05-12T09:27:46.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:27:46.069 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-05-12T09:27:46.070 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:27:46.070 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:27:46.070 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:27:46.070 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:27:46.071 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:27:46.179 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:27:46.179 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:27:46.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:27:46.180 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:27:46.181 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:27:46.181 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:27:46.182 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:27:46.183 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:27:46.183 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:27:46.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:27:46.184 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:27:46.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:27:46.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:27:46.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-05-12T09:27:46.223 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:27:46.223 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:27:46.223 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148052 2021-05-12T09:27:46.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.148052 2021-05-12T09:27:46.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:27:46.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:27:46.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T09:27:46.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-dup 0 2021-05-12T09:27:46.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-05-12T09:27:46.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T09:27:46.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-05-12T09:27:46.227 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:27:46.227 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:27:46.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:27:46.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:27:46.229 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:27:46.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:27:46.230 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:27:46.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:27:46.231 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:27:46.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:27:46.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:27:46.232 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:27:46.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:27:46.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:27:46.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:27:46.232 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:27:46.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:27:46.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T09:27:46.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-05-12T09:27:46.234 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:27:46.235 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:27:46.235 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.148052 2021-05-12T09:27:46.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.148052 2021-05-12T09:27:46.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:27:46.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:27:46.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T09:27:46.237 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T09:27:46.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T09:27:46.303 INFO:tasks.workunit:Running workunit osd/osd-fast-mark-down.sh... 2021-05-12T09:27:46.303 DEBUG:teuthology.orchestra.run.smithi028:workunit test osd/osd-fast-mark-down.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2021-05-12T09:27:46.365 INFO:tasks.workunit.client.0.smithi028.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T09:27:46.372 INFO:tasks.workunit.client.0.smithi028.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T09:27:46.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:27:46.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:27:46.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T09:27:46.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T09:27:46.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T09:27:46.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T09:27:46.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T09:27:46.374 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T09:27:46.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-fast-mark-down 2021-05-12T09:27:46.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:23: run: local dir=td/osd-fast-mark-down 2021-05-12T09:27:46.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:24: run: shift 2021-05-12T09:27:46.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:25: run: rm -f 'td/osd-fast-mark-down/*.pid' 2021-05-12T09:27:46.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: export CEPH_MON=127.0.0.1:7126 2021-05-12T09:27:46.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: CEPH_MON=127.0.0.1:7126 2021-05-12T09:27:46.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:27: run: export CEPH_ARGS 2021-05-12T09:27:46.376 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: uuidgen 2021-05-12T09:27:46.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: CEPH_ARGS+='--fsid=ce5b8dcc-fcfe-4fde-aeed-7926fc134e8d --auth-supported=none ' 2021-05-12T09:27:46.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:30: run: OLD_ARGS='--fsid=ce5b8dcc-fcfe-4fde-aeed-7926fc134e8d --auth-supported=none ' 2021-05-12T09:27:46.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:31: run: CEPH_ARGS+='--osd-fast-fail-on-connection-refused=false ' 2021-05-12T09:27:46.378 INFO:tasks.workunit.client.0.smithi028.stdout:Ensuring old behavior is there... 2021-05-12T09:27:46.382 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:32: run: echo 'Ensuring old behavior is there...' 2021-05-12T09:27:46.382 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:33: run: test_fast_kill td/osd-fast-mark-down 2021-05-12T09:27:46.382 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2021-05-12T09:27:46.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2021-05-12T09:27:46.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2021-05-12T09:27:46.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-05-12T09:27:46.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:27:46.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-05-12T09:27:46.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:27:46.385 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:27:46.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:27:46.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:27:46.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:27:46.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:27:46.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:27:46.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:27:46.387 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:27:46.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:27:46.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:27:46.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:27:46.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:27:46.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:27:46.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:27:46.389 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:27:46.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:27:46.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:27:46.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-05-12T09:27:46.389 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:27:46.390 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:27:46.390 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151428 2021-05-12T09:27:46.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.151428 2021-05-12T09:27:46.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:27:46.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:27:46.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2021-05-12T09:27:46.393 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:27:46.393 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:27:46.393 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151428 2021-05-12T09:27:46.394 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.151428 2021-05-12T09:27:46.394 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:27:46.394 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T09:27:46.395 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T09:27:46.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:27:46.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2021-05-12T09:27:46.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2021-05-12T09:27:46.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2021-05-12T09:27:46.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:27:46.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:27:46.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:27:46.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2021-05-12T09:27:46.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2021-05-12T09:27:46.404 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:46.401+0000 7fc4b4ceb700 -1 failed for service _ceph-mon._tcp 2021-05-12T09:27:46.404 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:46.401+0000 7fc4b4ceb700 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2021-05-12T09:27:46.404 INFO:tasks.workunit.client.0.smithi028.stderr: 2021-05-12T09:27:46.405 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:46.401+0000 7fc4b4ceb700 -1 ceph-mon: warning: no initial monitors; must use admin socket to feed hints 2021-05-12T09:27:46.747 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:27:46.748 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:27:46.748 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:27:46.748 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:27:46.748 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:27:46.748 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151428 2021-05-12T09:27:46.749 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151428/$cluster-$name.asok' 2021-05-12T09:27:46.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.151428/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2021-05-12T09:27:46.987 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:46.984+0000 7fc01211e700 -1 failed for service _ceph-mon._tcp 2021-05-12T09:27:46.987 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:46.984+0000 7fc01211e700 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2021-05-12T09:27:46.988 INFO:tasks.workunit.client.0.smithi028.stderr: 2021-05-12T09:27:46.988 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:46.984+0000 7fc01211e700 -1 ceph-mon: error generating initial monmap: (2) No such file or directory 2021-05-12T09:27:46.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: return 1 2021-05-12T09:27:46.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: return 1 2021-05-12T09:27:46.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:35: run: CEPH_ARGS='--fsid=ce5b8dcc-fcfe-4fde-aeed-7926fc134e8d --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2021-05-12T09:27:46.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:36: run: OLD_ARGS='--fsid=ce5b8dcc-fcfe-4fde-aeed-7926fc134e8d --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2021-05-12T09:27:46.992 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:38: run: CEPH_ARGS='--fsid=ce5b8dcc-fcfe-4fde-aeed-7926fc134e8d --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-05-12T09:27:46.992 INFO:tasks.workunit.client.0.smithi028.stdout:Testing async msgr... 2021-05-12T09:27:46.994 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:39: run: echo 'Testing async msgr...' 2021-05-12T09:27:46.994 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:40: run: test_fast_kill td/osd-fast-mark-down 2021-05-12T09:27:46.994 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2021-05-12T09:27:46.995 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2021-05-12T09:27:46.995 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2021-05-12T09:27:46.995 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-05-12T09:27:46.995 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:27:46.995 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-05-12T09:27:46.996 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:27:46.996 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:27:46.996 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:27:46.996 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:27:46.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:27:46.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:27:46.997 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:27:46.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:27:46.997 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:27:46.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:27:46.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:27:46.999 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:27:46.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:27:47.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:27:47.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:27:47.000 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:27:47.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:27:47.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:27:47.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-05-12T09:27:47.002 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:27:47.002 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:27:47.003 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151428 2021-05-12T09:27:47.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.151428 2021-05-12T09:27:47.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:27:47.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:27:47.004 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2021-05-12T09:27:47.004 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:27:47.004 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:27:47.004 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151428 2021-05-12T09:27:47.004 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.151428 2021-05-12T09:27:47.006 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:27:47.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T09:27:47.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:27:47.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2021-05-12T09:27:47.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2021-05-12T09:27:47.008 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2021-05-12T09:27:47.008 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:27:47.008 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:27:47.008 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:27:47.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2021-05-12T09:27:47.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2021-05-12T09:27:47.360 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:27:47.361 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:27:47.361 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:27:47.361 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:27:47.361 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:27:47.362 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151428 2021-05-12T09:27:47.362 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151428/$cluster-$name.asok' 2021-05-12T09:27:47.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.151428/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2021-05-12T09:27:47.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:27:47.647 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:27:47.647 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:27:47.647 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:27:47.647 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:27:47.648 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:27:47.648 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:27:47.648 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:27:47.648 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:27:47.649 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:27:47.649 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:27:47.650 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151428 2021-05-12T09:27:47.650 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.151428/ceph-mon.a.asok 2021-05-12T09:27:47.650 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:27:47.650 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.151428/ceph-mon.a.asok config get fsid 2021-05-12T09:27:47.757 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:27:47.757 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:27:47.757 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:27:47.758 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:27:47.758 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:27:47.758 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:27:47.758 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:27:47.759 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:27:47.759 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:27:47.759 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:27:47.760 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151428 2021-05-12T09:27:47.760 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.151428/ceph-mon.a.asok 2021-05-12T09:27:47.760 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:27:47.760 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.151428/ceph-mon.a.asok config get mon_host 2021-05-12T09:27:47.857 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:50: test_fast_kill: run_mgr td/osd-fast-mark-down x 2021-05-12T09:27:47.858 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-fast-mark-down 2021-05-12T09:27:47.858 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:27:47.858 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:27:47.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:27:47.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-fast-mark-down/x 2021-05-12T09:27:47.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:27:48.349 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:27:48.350 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:27:48.350 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:27:48.350 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:27:48.350 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:27:48.351 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151428 2021-05-12T09:27:48.351 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151428/$cluster-$name.asok' 2021-05-12T09:27:48.351 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:27:48.352 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-fast-mark-down/x '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.151428/$cluster-$name.asok' --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:27:48.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-05-12T09:27:48.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 0 2021-05-12T09:27:48.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2021-05-12T09:27:48.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:27:48.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:27:48.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:27:48.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/0 2021-05-12T09:27:48.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ce5b8dcc-fcfe-4fde-aeed-7926fc134e8d --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-05-12T09:27:48.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:27:48.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:27:48.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:27:48.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/0' 2021-05-12T09:27:48.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/0/journal' 2021-05-12T09:27:48.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:27:48.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:27:48.382 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-05-12T09:27:48.382 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:27:48.382 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:27:48.382 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:27:48.383 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:27:48.383 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:27:48.383 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151428 2021-05-12T09:27:48.383 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151428/$cluster-$name.asok' 2021-05-12T09:27:48.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.151428/$cluster-$name.asok' 2021-05-12T09:27:48.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:27:48.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:27:48.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:27:48.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-05-12T09:27:48.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-05-12T09:27:48.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:27:48.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:27:48.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:27:48.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:27:48.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:27:48.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-fast-mark-down/0 2021-05-12T09:27:48.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:27:48.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=d5ccea8e-4c31-44ca-b73d-e174fd8facf1 2021-05-12T09:27:48.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 d5ccea8e-4c31-44ca-b73d-e174fd8facf1' 2021-05-12T09:27:48.387 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 d5ccea8e-4c31-44ca-b73d-e174fd8facf1 2021-05-12T09:27:48.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:27:48.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCUn5tgSVNWFxAA3JdrN37vPWdm9t+da2sgAA== 2021-05-12T09:27:48.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCUn5tgSVNWFxAA3JdrN37vPWdm9t+da2sgAA=="}' 2021-05-12T09:27:48.394 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new d5ccea8e-4c31-44ca-b73d-e174fd8facf1 -i td/osd-fast-mark-down/0/new.json 2021-05-12T09:27:48.712 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:27:48.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-fast-mark-down/0/new.json 2021-05-12T09:27:48.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=ce5b8dcc-fcfe-4fde-aeed-7926fc134e8d --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.151428/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCUn5tgSVNWFxAA3JdrN37vPWdm9t+da2sgAA== --osd-uuid d5ccea8e-4c31-44ca-b73d-e174fd8facf1 2021-05-12T09:27:48.749 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:48.746+0000 7fb0fe15c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:27:48.751 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:48.748+0000 7fb0fe15c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:27:48.752 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:48.750+0000 7fb0fe15c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:27:48.753 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:48.750+0000 7fb0fe15c0c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-05-12T09:27:48.753 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:48.750+0000 7fb0fe15c0c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-05-12T09:27:48.754 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:48.750+0000 7fb0fe15c0c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-05-12T09:27:48.792 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:48.790+0000 7fb0fe15c0c0 -1 bluestore(td/osd-fast-mark-down/0) _read_fsid unparsable uuid 2021-05-12T09:27:53.410 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-fast-mark-down/0/keyring 2021-05-12T09:27:53.410 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:27:53.411 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:27:53.411 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:27:53.411 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-fast-mark-down/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:27:53.913 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:27:53.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:27:53.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:27:53.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=ce5b8dcc-fcfe-4fde-aeed-7926fc134e8d --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.151428/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:27:53.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:27:53.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:27:53.936 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:53.934+0000 7ff4983c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:27:53.938 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:53.936+0000 7ff4983c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:27:53.940 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:53.937+0000 7ff4983c50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:27:54.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:27:54.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:27:54.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:27:54.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:27:54.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:27:54.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:27:54.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:27:54.298 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:27:54.298 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:27:54.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:27:54.460 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:54.458+0000 7ff4983c50c0 -1 Falling back to public interface 2021-05-12T09:27:54.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:27:55.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:27:55.695 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:27:55.695 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:27:55.696 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:27:55.697 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:27:55.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:27:56.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:27:56.636 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:56.633+0000 7ff4983c50c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:27:57.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:27:57.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:27:57.088 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:27:57.088 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:27:57.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:27:57.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:27:57.502 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/152613,v1:127.0.0.1:6803/152613] [v2:127.0.0.1:6804/152613,v1:127.0.0.1:6805/152613] exists,up d5ccea8e-4c31-44ca-b73d-e174fd8facf1 2021-05-12T09:27:57.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:27:57.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:27:57.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:27:57.508 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.0.pid 2021-05-12T09:27:57.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=152613 2021-05-12T09:27:57.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-05-12T09:27:57.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 1 2021-05-12T09:27:57.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2021-05-12T09:27:57.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:27:57.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:27:57.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:27:57.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/1 2021-05-12T09:27:57.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ce5b8dcc-fcfe-4fde-aeed-7926fc134e8d --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-05-12T09:27:57.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:27:57.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:27:57.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:27:57.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/1' 2021-05-12T09:27:57.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/1/journal' 2021-05-12T09:27:57.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:27:57.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:27:57.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-05-12T09:27:57.512 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:27:57.513 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:27:57.513 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:27:57.513 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:27:57.513 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:27:57.514 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151428 2021-05-12T09:27:57.514 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151428/$cluster-$name.asok' 2021-05-12T09:27:57.514 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.151428/$cluster-$name.asok' 2021-05-12T09:27:57.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:27:57.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:27:57.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:27:57.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-05-12T09:27:57.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-05-12T09:27:57.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:27:57.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:27:57.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:27:57.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:27:57.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:27:57.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-fast-mark-down/1 2021-05-12T09:27:57.518 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:27:57.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=f15e300d-4ff8-44cf-a9a3-9fa481a67618 2021-05-12T09:27:57.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 f15e300d-4ff8-44cf-a9a3-9fa481a67618' 2021-05-12T09:27:57.519 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 f15e300d-4ff8-44cf-a9a3-9fa481a67618 2021-05-12T09:27:57.519 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:27:57.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCdn5tgrMv9HhAAqoXyNhJDcEVkGuplqCzBOQ== 2021-05-12T09:27:57.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCdn5tgrMv9HhAAqoXyNhJDcEVkGuplqCzBOQ=="}' 2021-05-12T09:27:57.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new f15e300d-4ff8-44cf-a9a3-9fa481a67618 -i td/osd-fast-mark-down/1/new.json 2021-05-12T09:27:58.018 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:27:58.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-fast-mark-down/1/new.json 2021-05-12T09:27:58.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=ce5b8dcc-fcfe-4fde-aeed-7926fc134e8d --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.151428/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCdn5tgrMv9HhAAqoXyNhJDcEVkGuplqCzBOQ== --osd-uuid f15e300d-4ff8-44cf-a9a3-9fa481a67618 2021-05-12T09:27:58.056 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:58.054+0000 7f6a3f62f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:27:58.057 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:58.056+0000 7f6a3f62f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:27:58.059 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:58.057+0000 7f6a3f62f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:27:58.060 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:58.058+0000 7f6a3f62f0c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-05-12T09:27:58.060 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:58.058+0000 7f6a3f62f0c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-05-12T09:27:58.060 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:58.058+0000 7f6a3f62f0c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-05-12T09:27:58.114 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:27:58.112+0000 7f6a3f62f0c0 -1 bluestore(td/osd-fast-mark-down/1) _read_fsid unparsable uuid 2021-05-12T09:28:02.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-fast-mark-down/1/keyring 2021-05-12T09:28:02.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:28:02.923 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:28:02.924 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:28:02.924 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-fast-mark-down/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:28:03.445 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:28:03.446 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:28:03.447 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=ce5b8dcc-fcfe-4fde-aeed-7926fc134e8d --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.151428/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:28:03.447 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:28:03.447 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:28:03.447 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:28:03.469 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:03.467+0000 7feb348070c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:03.470 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:03.469+0000 7feb348070c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:03.472 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:03.470+0000 7feb348070c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:03.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:28:03.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:28:03.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:28:03.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:28:03.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:28:03.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:03.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:28:03.836 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:28:03.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:03.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:28:04.008 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:04.006+0000 7feb348070c0 -1 Falling back to public interface 2021-05-12T09:28:04.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:28:05.229 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:28:05.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:05.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:28:05.230 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:28:05.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:05.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:28:05.630 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:28:06.175 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:06.173+0000 7feb348070c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:28:06.631 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:28:06.632 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:06.632 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:28:06.633 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:28:06.633 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:06.634 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:28:07.042 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/154007,v1:127.0.0.1:6811/154007] [v2:127.0.0.1:6812/154007,v1:127.0.0.1:6813/154007] exists,up f15e300d-4ff8-44cf-a9a3-9fa481a67618 2021-05-12T09:28:07.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:28:07.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:28:07.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:28:07.045 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.1.pid 2021-05-12T09:28:07.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=154007 2021-05-12T09:28:07.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-05-12T09:28:07.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 2 2021-05-12T09:28:07.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2021-05-12T09:28:07.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:28:07.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T09:28:07.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:28:07.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/2 2021-05-12T09:28:07.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ce5b8dcc-fcfe-4fde-aeed-7926fc134e8d --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-05-12T09:28:07.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:28:07.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:28:07.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:28:07.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/2' 2021-05-12T09:28:07.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/2/journal' 2021-05-12T09:28:07.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:28:07.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:28:07.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-05-12T09:28:07.054 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:28:07.054 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:28:07.055 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:28:07.055 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:28:07.055 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:28:07.055 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151428 2021-05-12T09:28:07.056 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151428/$cluster-$name.asok' 2021-05-12T09:28:07.056 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.151428/$cluster-$name.asok' 2021-05-12T09:28:07.056 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:28:07.056 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:28:07.056 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:28:07.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-05-12T09:28:07.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-05-12T09:28:07.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:28:07.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:28:07.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:28:07.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:28:07.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:28:07.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-fast-mark-down/2 2021-05-12T09:28:07.059 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:28:07.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=766c8ca3-eb79-4ad0-b20e-80e4ce9d212a 2021-05-12T09:28:07.059 INFO:tasks.workunit.client.0.smithi028.stdout:add osd2 766c8ca3-eb79-4ad0-b20e-80e4ce9d212a 2021-05-12T09:28:07.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 766c8ca3-eb79-4ad0-b20e-80e4ce9d212a' 2021-05-12T09:28:07.060 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:28:07.065 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCnn5tg5VzHAxAAR51hHDQ1praz9VEPYQI3bA== 2021-05-12T09:28:07.065 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCnn5tg5VzHAxAAR51hHDQ1praz9VEPYQI3bA=="}' 2021-05-12T09:28:07.066 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 766c8ca3-eb79-4ad0-b20e-80e4ce9d212a -i td/osd-fast-mark-down/2/new.json 2021-05-12T09:28:07.631 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:28:07.641 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-fast-mark-down/2/new.json 2021-05-12T09:28:07.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=ce5b8dcc-fcfe-4fde-aeed-7926fc134e8d --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.151428/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCnn5tg5VzHAxAAR51hHDQ1praz9VEPYQI3bA== --osd-uuid 766c8ca3-eb79-4ad0-b20e-80e4ce9d212a 2021-05-12T09:28:07.667 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:07.665+0000 7f1a94a6e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:07.669 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:07.667+0000 7f1a94a6e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:07.671 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:07.668+0000 7f1a94a6e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:07.671 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:07.669+0000 7f1a94a6e0c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-05-12T09:28:07.671 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:07.669+0000 7f1a94a6e0c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-05-12T09:28:07.671 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:07.669+0000 7f1a94a6e0c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-05-12T09:28:07.711 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:07.708+0000 7f1a94a6e0c0 -1 bluestore(td/osd-fast-mark-down/2) _read_fsid unparsable uuid 2021-05-12T09:28:12.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-fast-mark-down/2/keyring 2021-05-12T09:28:12.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:28:12.477 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd2 key to auth repository 2021-05-12T09:28:12.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T09:28:12.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-fast-mark-down/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:28:12.990 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:28:12.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T09:28:12.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=ce5b8dcc-fcfe-4fde-aeed-7926fc134e8d --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.151428/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:28:12.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:28:12.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:28:12.992 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:28:13.013 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:13.011+0000 7fe1e3b5d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:13.015 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:13.013+0000 7fe1e3b5d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:13.016 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:13.015+0000 7fe1e3b5d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:13.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T09:28:13.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:28:13.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:28:13.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:28:13.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:28:13.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:13.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:28:13.398 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:28:13.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:13.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:28:13.534 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:13.532+0000 7fe1e3b5d0c0 -1 Falling back to public interface 2021-05-12T09:28:13.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:28:14.779 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:28:14.779 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:14.780 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:28:14.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:28:14.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:14.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:28:15.170 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:28:15.837 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:15.834+0000 7fe1e3b5d0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T09:28:16.171 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:28:16.172 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:16.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:28:16.173 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:28:16.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:16.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:28:16.561 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/155393,v1:127.0.0.1:6819/155393] [v2:127.0.0.1:6820/155393,v1:127.0.0.1:6821/155393] exists,up 766c8ca3-eb79-4ad0-b20e-80e4ce9d212a 2021-05-12T09:28:16.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:28:16.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:28:16.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:28:16.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.2.pid 2021-05-12T09:28:16.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=155393 2021-05-12T09:28:16.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:56: test_fast_kill: create_rbd_pool 2021-05-12T09:28:16.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T09:28:17.042 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'rbd' does not exist 2021-05-12T09:28:17.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T09:28:17.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T09:28:17.652 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'rbd' already exists 2021-05-12T09:28:17.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:28:18.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T09:28:20.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:59: test_fast_kill: timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup 2021-05-12T09:28:20.475 INFO:tasks.workunit.client.0.smithi028.stdout:hints = 1 2021-05-12T09:28:20.476 INFO:tasks.workunit.client.0.smithi028.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 128 objects 2021-05-12T09:28:20.476 INFO:tasks.workunit.client.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_156006 2021-05-12T09:28:20.476 INFO:tasks.workunit.client.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-05-12T09:28:20.477 INFO:tasks.workunit.client.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2021-05-12T09:28:21.475 INFO:tasks.workunit.client.0.smithi028.stdout: 1 16 65 49 0.191383 0.191406 0.301774 0.268073 2021-05-12T09:28:22.476 INFO:tasks.workunit.client.0.smithi028.stdout: 2 16 113 97 0.189426 0.1875 0.283602 0.301346 2021-05-12T09:28:23.476 INFO:tasks.workunit.client.0.smithi028.stdout:Total time run: 2.3827 2021-05-12T09:28:23.476 INFO:tasks.workunit.client.0.smithi028.stdout:Total writes made: 128 2021-05-12T09:28:23.477 INFO:tasks.workunit.client.0.smithi028.stdout:Write size: 4096 2021-05-12T09:28:23.477 INFO:tasks.workunit.client.0.smithi028.stdout:Object size: 4096 2021-05-12T09:28:23.477 INFO:tasks.workunit.client.0.smithi028.stdout:Bandwidth (MB/sec): 0.209846 2021-05-12T09:28:23.477 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev Bandwidth: 0.00276214 2021-05-12T09:28:23.477 INFO:tasks.workunit.client.0.smithi028.stdout:Max bandwidth (MB/sec): 0.191406 2021-05-12T09:28:23.479 INFO:tasks.workunit.client.0.smithi028.stdout:Min bandwidth (MB/sec): 0.1875 2021-05-12T09:28:23.479 INFO:tasks.workunit.client.0.smithi028.stdout:Average IOPS: 53 2021-05-12T09:28:23.479 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev IOPS: 0.707107 2021-05-12T09:28:23.480 INFO:tasks.workunit.client.0.smithi028.stdout:Max IOPS: 49 2021-05-12T09:28:23.480 INFO:tasks.workunit.client.0.smithi028.stdout:Min IOPS: 48 2021-05-12T09:28:23.480 INFO:tasks.workunit.client.0.smithi028.stdout:Average Latency(s): 0.296496 2021-05-12T09:28:23.480 INFO:tasks.workunit.client.0.smithi028.stdout:Stddev Latency(s): 0.0667955 2021-05-12T09:28:23.480 INFO:tasks.workunit.client.0.smithi028.stdout:Max latency(s): 0.452813 2021-05-12T09:28:23.481 INFO:tasks.workunit.client.0.smithi028.stdout:Min latency(s): 0.158264 2021-05-12T09:28:23.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:60: test_fast_kill: sleep 1 2021-05-12T09:28:24.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:62: test_fast_kill: killid=0 2021-05-12T09:28:24.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:63: test_fast_kill: previd=0 2021-05-12T09:28:24.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2021-05-12T09:28:24.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 0 -eq 0 ']' 2021-05-12T09:28:24.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=155393 2021-05-12T09:28:24.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 155393 -eq 0 ']' 2021-05-12T09:28:24.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=155393 2021-05-12T09:28:24.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 155393 2021-05-12T09:28:24.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2021-05-12T09:28:24.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2021-05-12T09:28:24.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2021-05-12T09:28:24.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2021-05-12T09:28:25.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2021-05-12T09:28:25.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2021-05-12T09:28:25.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2021-05-12T09:28:25.684 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2021-05-12T09:28:25.684 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2021-05-12T09:28:26.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=1 2021-05-12T09:28:26.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 1 -lt 1 ']' 2021-05-12T09:28:26.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 1 -gt 1 ']' 2021-05-12T09:28:26.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2021-05-12T09:28:26.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 1 -lt 1 ']' 2021-05-12T09:28:26.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2021-05-12T09:28:26.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 155393 -eq 155393 ']' 2021-05-12T09:28:26.079 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=155393 2021-05-12T09:28:26.079 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 155393 -eq 155393 ']' 2021-05-12T09:28:26.079 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=152613 2021-05-12T09:28:26.080 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 152613 -eq 155393 ']' 2021-05-12T09:28:26.080 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=152613 2021-05-12T09:28:26.080 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 152613 2021-05-12T09:28:26.081 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2021-05-12T09:28:26.081 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2021-05-12T09:28:26.081 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2021-05-12T09:28:26.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2021-05-12T09:28:27.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2021-05-12T09:28:27.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2021-05-12T09:28:27.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2021-05-12T09:28:27.087 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2021-05-12T09:28:27.087 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2021-05-12T09:28:27.477 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=2 2021-05-12T09:28:27.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 2 -lt 2 ']' 2021-05-12T09:28:27.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 2 -gt 2 ']' 2021-05-12T09:28:27.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2021-05-12T09:28:27.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 2 -lt 2 ']' 2021-05-12T09:28:27.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:103: test_fast_kill: pkill -SIGTERM rados 2021-05-12T09:28:27.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:104: test_fast_kill: teardown td/osd-fast-mark-down 2021-05-12T09:28:27.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-05-12T09:28:27.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:28:27.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-05-12T09:28:27.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:28:27.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:28:27.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:28:27.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:28:27.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:28:27.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:28:27.607 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:28:27.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:28:27.609 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:28:27.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:28:27.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:28:27.610 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:28:27.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:28:27.612 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:28:27.612 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:28:27.612 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:28:27.613 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:28:27.614 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:28:27.614 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-05-12T09:28:27.621 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:28:27.622 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:28:27.622 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151428 2021-05-12T09:28:27.622 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.151428 2021-05-12T09:28:27.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:28:27.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:28:27.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:42: run: return 0 2021-05-12T09:28:27.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T09:28:27.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-fast-mark-down 0 2021-05-12T09:28:27.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-05-12T09:28:27.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T09:28:27.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-05-12T09:28:27.625 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:28:27.625 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:28:27.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:28:27.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:28:27.626 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:28:27.627 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:28:27.628 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:28:27.628 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:28:27.628 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:28:27.628 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:28:27.629 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:28:27.629 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:28:27.629 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:28:27.629 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:28:27.630 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:28:27.630 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:28:27.630 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:28:27.632 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T09:28:27.632 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-05-12T09:28:27.633 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:28:27.633 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:28:27.633 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151428 2021-05-12T09:28:27.633 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.151428 2021-05-12T09:28:27.634 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:28:27.634 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:28:27.634 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T09:28:27.634 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T09:28:27.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T09:28:27.698 INFO:tasks.workunit:Running workunit osd/osd-force-create-pg.sh... 2021-05-12T09:28:27.698 DEBUG:teuthology.orchestra.run.smithi028:workunit test osd/osd-force-create-pg.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2021-05-12T09:28:27.759 INFO:tasks.workunit.client.0.smithi028.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T09:28:27.766 INFO:tasks.workunit.client.0.smithi028.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T09:28:27.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:28:27.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:28:27.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T09:28:27.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T09:28:27.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T09:28:27.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T09:28:27.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T09:28:27.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T09:28:27.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-force-create-pg 2021-05-12T09:28:27.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:5: run: local dir=td/osd-force-create-pg 2021-05-12T09:28:27.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:6: run: shift 2021-05-12T09:28:27.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: export CEPH_MON=127.0.0.1:7145 2021-05-12T09:28:27.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: CEPH_MON=127.0.0.1:7145 2021-05-12T09:28:27.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:9: run: export CEPH_ARGS 2021-05-12T09:28:27.770 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: uuidgen 2021-05-12T09:28:27.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: CEPH_ARGS+='--fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none ' 2021-05-12T09:28:27.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7145 ' 2021-05-12T09:28:27.771 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: set 2021-05-12T09:28:27.771 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T09:28:27.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: local funcs=TEST_reuse_id 2021-05-12T09:28:27.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:14: run: for func in $funcs 2021-05-12T09:28:27.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:15: run: TEST_reuse_id td/osd-force-create-pg 2021-05-12T09:28:27.776 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:20: TEST_reuse_id: local dir=td/osd-force-create-pg 2021-05-12T09:28:27.776 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:22: TEST_reuse_id: setup td/osd-force-create-pg 2021-05-12T09:28:27.776 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-force-create-pg 2021-05-12T09:28:27.776 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-force-create-pg 2021-05-12T09:28:27.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2021-05-12T09:28:27.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:28:27.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2021-05-12T09:28:27.777 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:28:27.777 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:28:27.778 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:28:27.778 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:28:27.778 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:28:27.779 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:28:27.779 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:28:27.779 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:28:27.779 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:28:27.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:28:27.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:28:27.780 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:28:27.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:28:27.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:28:27.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:28:27.781 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:28:27.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:28:27.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:28:27.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2021-05-12T09:28:27.785 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:28:27.785 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:28:27.785 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156168 2021-05-12T09:28:27.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.156168 2021-05-12T09:28:27.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:28:27.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:28:27.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-force-create-pg 2021-05-12T09:28:27.786 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:28:27.787 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:28:27.787 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156168 2021-05-12T09:28:27.787 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.156168 2021-05-12T09:28:27.787 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:28:27.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T09:28:27.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T09:28:27.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:28:27.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-force-create-pg 1' TERM HUP INT 2021-05-12T09:28:27.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:23: TEST_reuse_id: run_mon td/osd-force-create-pg a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-05-12T09:28:27.789 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-force-create-pg 2021-05-12T09:28:27.789 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:28:27.789 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:28:27.789 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:28:27.790 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-force-create-pg/a 2021-05-12T09:28:27.790 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-force-create-pg/a --run-dir=td/osd-force-create-pg --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-05-12T09:28:28.126 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:28:28.127 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:28:28.128 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:28:28.128 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:28:28.128 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:28:28.128 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156168 2021-05-12T09:28:28.129 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156168/$cluster-$name.asok' 2021-05-12T09:28:28.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-force-create-pg/a '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' --mon-cluster-log-file=td/osd-force-create-pg/log --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-05-12T09:28:28.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:28:28.421 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:28:28.421 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:28:28.421 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:28:28.421 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:28:28.422 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:28:28.422 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:28:28.422 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:28:28.422 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:28:28.423 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:28:28.423 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:28:28.423 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156168 2021-05-12T09:28:28.423 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.156168/ceph-mon.a.asok 2021-05-12T09:28:28.423 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:28:28.424 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.156168/ceph-mon.a.asok config get fsid 2021-05-12T09:28:28.525 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:28:28.526 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:28:28.526 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:28:28.526 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:28:28.526 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:28:28.527 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:28:28.527 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:28:28.527 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:28:28.527 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:28:28.528 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:28:28.528 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156168 2021-05-12T09:28:28.528 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.156168/ceph-mon.a.asok 2021-05-12T09:28:28.528 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:28:28.529 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.156168/ceph-mon.a.asok config get mon_host 2021-05-12T09:28:28.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:24: TEST_reuse_id: run_mgr td/osd-force-create-pg x 2021-05-12T09:28:28.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-force-create-pg 2021-05-12T09:28:28.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:28:28.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:28:28.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:28:28.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-force-create-pg/x 2021-05-12T09:28:28.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:28:29.123 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:28:29.124 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:28:29.124 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:28:29.124 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:28:29.124 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:28:29.125 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156168 2021-05-12T09:28:29.125 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156168/$cluster-$name.asok' 2021-05-12T09:28:29.125 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:28:29.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-force-create-pg/x '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:28:29.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:25: TEST_reuse_id: run_osd td/osd-force-create-pg 0 2021-05-12T09:28:29.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2021-05-12T09:28:29.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:28:29.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:28:29.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:28:29.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/0 2021-05-12T09:28:29.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T09:28:29.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:28:29.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:28:29.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:28:29.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2021-05-12T09:28:29.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2021-05-12T09:28:29.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:28:29.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:28:29.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T09:28:29.154 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:28:29.154 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:28:29.155 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:28:29.155 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:28:29.155 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:28:29.155 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156168 2021-05-12T09:28:29.156 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156168/$cluster-$name.asok' 2021-05-12T09:28:29.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' 2021-05-12T09:28:29.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:28:29.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:28:29.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:28:29.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T09:28:29.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T09:28:29.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:28:29.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:28:29.158 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:28:29.158 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:28:29.158 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:28:29.158 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-force-create-pg/0 2021-05-12T09:28:29.159 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:28:29.159 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=9592a29b-2fe3-4bbc-ace2-fd9908c5559c 2021-05-12T09:28:29.159 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 9592a29b-2fe3-4bbc-ace2-fd9908c5559c' 2021-05-12T09:28:29.159 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 9592a29b-2fe3-4bbc-ace2-fd9908c5559c 2021-05-12T09:28:29.160 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:28:29.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQC9n5tgb/SbCRAA1gqTdBqTNF4CQeg5N1wKBA== 2021-05-12T09:28:29.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQC9n5tgb/SbCRAA1gqTdBqTNF4CQeg5N1wKBA=="}' 2021-05-12T09:28:29.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 9592a29b-2fe3-4bbc-ace2-fd9908c5559c -i td/osd-force-create-pg/0/new.json 2021-05-12T09:28:29.502 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:28:29.513 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-force-create-pg/0/new.json 2021-05-12T09:28:29.513 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC9n5tgb/SbCRAA1gqTdBqTNF4CQeg5N1wKBA== --osd-uuid 9592a29b-2fe3-4bbc-ace2-fd9908c5559c 2021-05-12T09:28:29.535 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:29.532+0000 7f4fd71a80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:29.537 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:29.534+0000 7f4fd71a80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:29.538 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:29.536+0000 7f4fd71a80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:29.540 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:29.537+0000 7f4fd71a80c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-05-12T09:28:29.541 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:29.537+0000 7f4fd71a80c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-05-12T09:28:29.541 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:29.537+0000 7f4fd71a80c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-05-12T09:28:29.575 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:29.572+0000 7f4fd71a80c0 -1 bluestore(td/osd-force-create-pg/0) _read_fsid unparsable uuid 2021-05-12T09:28:34.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-force-create-pg/0/keyring 2021-05-12T09:28:34.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:28:34.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:28:34.225 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:28:34.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-force-create-pg/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:28:34.721 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:28:34.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:28:34.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:28:34.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:28:34.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:28:34.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:28:34.741 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:34.739+0000 7f5eb1d320c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:34.743 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:34.741+0000 7f5eb1d320c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:34.744 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:34.743+0000 7f5eb1d320c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:35.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:28:35.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:28:35.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:28:35.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:28:35.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:28:35.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:35.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:28:35.098 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:28:35.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:35.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:28:35.264 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:35.262+0000 7f5eb1d320c0 -1 Falling back to public interface 2021-05-12T09:28:35.493 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:28:36.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:28:36.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:36.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:28:36.495 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:28:36.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:36.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:28:36.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:28:37.501 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:37.499+0000 7f5eb1d320c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:28:37.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:28:37.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:37.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:28:37.886 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:28:37.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:37.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:28:38.277 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/157288,v1:127.0.0.1:6803/157288] [v2:127.0.0.1:6804/157288,v1:127.0.0.1:6805/157288] exists,up 9592a29b-2fe3-4bbc-ace2-fd9908c5559c 2021-05-12T09:28:38.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:28:38.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:28:38.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:28:38.284 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:26: TEST_reuse_id: run_osd td/osd-force-create-pg 1 2021-05-12T09:28:38.284 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2021-05-12T09:28:38.284 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:28:38.285 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:28:38.285 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:28:38.285 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/1 2021-05-12T09:28:38.286 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T09:28:38.286 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:28:38.286 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:28:38.286 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:28:38.287 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2021-05-12T09:28:38.287 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2021-05-12T09:28:38.287 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:28:38.287 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:28:38.287 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T09:28:38.288 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:28:38.288 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:28:38.288 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:28:38.289 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:28:38.289 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:28:38.289 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156168 2021-05-12T09:28:38.289 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156168/$cluster-$name.asok' 2021-05-12T09:28:38.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' 2021-05-12T09:28:38.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:28:38.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:28:38.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:28:38.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T09:28:38.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T09:28:38.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:28:38.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:28:38.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:28:38.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:28:38.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:28:38.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-force-create-pg/1 2021-05-12T09:28:38.293 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:28:38.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=657ba8d3-e8fb-467b-a80b-a3785bc02451 2021-05-12T09:28:38.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 657ba8d3-e8fb-467b-a80b-a3785bc02451' 2021-05-12T09:28:38.294 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 657ba8d3-e8fb-467b-a80b-a3785bc02451 2021-05-12T09:28:38.295 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:28:38.295 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDGn5tg8XtkERAAgglCBKZ+UtoBGuxG3YYYNw== 2021-05-12T09:28:38.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDGn5tg8XtkERAAgglCBKZ+UtoBGuxG3YYYNw=="}' 2021-05-12T09:28:38.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 657ba8d3-e8fb-467b-a80b-a3785bc02451 -i td/osd-force-create-pg/1/new.json 2021-05-12T09:28:38.856 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:28:38.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-force-create-pg/1/new.json 2021-05-12T09:28:38.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDGn5tg8XtkERAAgglCBKZ+UtoBGuxG3YYYNw== --osd-uuid 657ba8d3-e8fb-467b-a80b-a3785bc02451 2021-05-12T09:28:38.891 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:38.888+0000 7fbd86c680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:38.892 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:38.890+0000 7fbd86c680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:38.894 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:38.892+0000 7fbd86c680c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:38.895 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:38.893+0000 7fbd86c680c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-05-12T09:28:38.895 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:38.893+0000 7fbd86c680c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-05-12T09:28:38.895 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:38.893+0000 7fbd86c680c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-05-12T09:28:38.954 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:38.952+0000 7fbd86c680c0 -1 bluestore(td/osd-force-create-pg/1) _read_fsid unparsable uuid 2021-05-12T09:28:43.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-force-create-pg/1/keyring 2021-05-12T09:28:43.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:28:43.823 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:28:43.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:28:43.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-force-create-pg/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:28:44.334 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:28:44.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:28:44.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:28:44.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:28:44.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:28:44.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:28:44.354 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:44.352+0000 7fb6e2d020c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:44.378 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:44.376+0000 7fb6e2d020c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:44.379 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:44.377+0000 7fb6e2d020c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:28:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:28:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:28:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:28:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:28:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:28:44.712 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:28:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:28:44.896 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:44.894+0000 7fb6e2d020c0 -1 Falling back to public interface 2021-05-12T09:28:45.107 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:28:46.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:28:46.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:46.110 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:28:46.111 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:28:46.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:46.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:28:46.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:28:47.098 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:47.096+0000 7fb6e2d020c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:28:47.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:28:47.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:47.516 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:28:47.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:28:47.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:47.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:28:47.904 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/158673,v1:127.0.0.1:6811/158673] [v2:127.0.0.1:6812/158673,v1:127.0.0.1:6813/158673] exists,up 657ba8d3-e8fb-467b-a80b-a3785bc02451 2021-05-12T09:28:47.908 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:28:47.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:28:47.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:28:47.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:27: TEST_reuse_id: run_osd td/osd-force-create-pg 2 2021-05-12T09:28:47.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2021-05-12T09:28:47.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:28:47.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T09:28:47.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:28:47.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/2 2021-05-12T09:28:47.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T09:28:47.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:28:47.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:28:47.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:28:47.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2021-05-12T09:28:47.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2021-05-12T09:28:47.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:28:47.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:28:47.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T09:28:47.913 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:28:47.913 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:28:47.913 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:28:47.914 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:28:47.914 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:28:47.914 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156168 2021-05-12T09:28:47.915 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156168/$cluster-$name.asok' 2021-05-12T09:28:47.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' 2021-05-12T09:28:47.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:28:47.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:28:47.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:28:47.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T09:28:47.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T09:28:47.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:28:47.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:28:47.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:28:47.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:28:47.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:28:47.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-force-create-pg/2 2021-05-12T09:28:47.918 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:28:47.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=7e8a2c8c-5f59-4812-8149-741863d734d5 2021-05-12T09:28:47.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 7e8a2c8c-5f59-4812-8149-741863d734d5' 2021-05-12T09:28:47.919 INFO:tasks.workunit.client.0.smithi028.stdout:add osd2 7e8a2c8c-5f59-4812-8149-741863d734d5 2021-05-12T09:28:47.920 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:28:47.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDPn5tgyFPjNhAAg5XRUflkfR6/M42aVRypHA== 2021-05-12T09:28:47.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDPn5tgyFPjNhAAg5XRUflkfR6/M42aVRypHA=="}' 2021-05-12T09:28:47.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 7e8a2c8c-5f59-4812-8149-741863d734d5 -i td/osd-force-create-pg/2/new.json 2021-05-12T09:28:48.471 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:28:48.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-force-create-pg/2/new.json 2021-05-12T09:28:48.484 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDPn5tgyFPjNhAAg5XRUflkfR6/M42aVRypHA== --osd-uuid 7e8a2c8c-5f59-4812-8149-741863d734d5 2021-05-12T09:28:48.505 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:48.504+0000 7f2c34a3f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:48.507 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:48.505+0000 7f2c34a3f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:48.508 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:48.506+0000 7f2c34a3f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:48.509 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:48.507+0000 7f2c34a3f0c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-05-12T09:28:48.509 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:48.507+0000 7f2c34a3f0c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-05-12T09:28:48.509 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:48.507+0000 7f2c34a3f0c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-05-12T09:28:48.593 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:48.591+0000 7f2c34a3f0c0 -1 bluestore(td/osd-force-create-pg/2) _read_fsid unparsable uuid 2021-05-12T09:28:53.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-force-create-pg/2/keyring 2021-05-12T09:28:53.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:28:53.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T09:28:53.301 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd2 key to auth repository 2021-05-12T09:28:53.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-force-create-pg/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:28:53.814 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:28:53.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T09:28:53.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:28:53.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:28:53.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:28:53.816 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:28:53.834 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:53.831+0000 7fd64805d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:53.835 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:53.833+0000 7fd64805d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:53.836 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:53.834+0000 7fd64805d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:28:54.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T09:28:54.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:28:54.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:28:54.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:28:54.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:28:54.202 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:54.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:28:54.203 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:28:54.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:54.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:28:54.361 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:54.359+0000 7fd64805d0c0 -1 Falling back to public interface 2021-05-12T09:28:54.593 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:28:55.595 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:28:55.595 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:55.596 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:28:55.597 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:28:55.598 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:55.598 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:28:55.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:28:56.614 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:28:56.611+0000 7fd64805d0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T09:28:56.980 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:28:56.981 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:28:56.981 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:28:56.982 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:28:56.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:28:56.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:28:57.371 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/160058,v1:127.0.0.1:6819/160058] [v2:127.0.0.1:6820/160058,v1:127.0.0.1:6821/160058] exists,up 7e8a2c8c-5f59-4812-8149-741863d734d5 2021-05-12T09:28:57.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:28:57.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:28:57.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:28:57.374 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:29: TEST_reuse_id: ceph osd pool create foo 50 2021-05-12T09:28:57.976 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'foo' already exists 2021-05-12T09:28:57.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:30: TEST_reuse_id: wait_for_clean 2021-05-12T09:28:57.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:28:57.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:28:57.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:28:57.987 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:28:57.987 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T09:28:57.988 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:28:57.988 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:28:57.989 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:28:57.989 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:28:57.989 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:28:58.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:28:58.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:28:58.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:28:58.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:28:58.078 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:28:58.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:28:58.625 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:28:58.625 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:28:58.626 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:28:58.626 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:28:58.626 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:28:58.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836487 2021-05-12T09:28:58.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836487 2021-05-12T09:28:58.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487' 2021-05-12T09:28:58.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:28:58.769 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:28:58.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672965 2021-05-12T09:28:58.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672965 2021-05-12T09:28:58.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2021-05-12T09:28:58.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:28:58.963 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:28:59.093 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=64424509443 2021-05-12T09:28:59.093 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 64424509443 2021-05-12T09:28:59.094 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509443' 2021-05-12T09:28:59.094 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:28:59.095 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-05-12T09:28:59.095 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:28:59.095 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:28:59.096 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836487 2021-05-12T09:28:59.096 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:28:59.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836487 2021-05-12T09:28:59.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-05-12T09:28:59.098 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836487 2021-05-12T09:28:59.098 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:28:59.656 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836485 -lt 21474836487 2021-05-12T09:28:59.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:29:00.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:29:00.658 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:29:01.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836488 -lt 21474836487 2021-05-12T09:29:01.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:29:01.048 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2021-05-12T09:29:01.048 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:29:01.049 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:29:01.050 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672965 2021-05-12T09:29:01.050 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:29:01.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672965 2021-05-12T09:29:01.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-05-12T09:29:01.051 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672965 2021-05-12T09:29:01.052 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:29:01.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672965 -lt 42949672965 2021-05-12T09:29:01.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:29:01.429 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509443 2021-05-12T09:29:01.429 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:29:01.429 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:29:01.431 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-64424509443 2021-05-12T09:29:01.431 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:29:01.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=64424509443 2021-05-12T09:29:01.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2021-05-12T09:29:01.433 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 64424509443 2021-05-12T09:29:01.433 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:29:01.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 64424509443 -lt 64424509443 2021-05-12T09:29:01.811 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:29:01.812 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:29:01.812 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:29:02.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 51 == 0 2021-05-12T09:29:02.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:29:02.303 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:29:02.303 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:29:02.303 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:29:02.303 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:29:02.304 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:29:02.304 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:29:02.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=37 2021-05-12T09:29:02.679 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:29:02.680 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:29:02.684 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:29:03.176 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 37 = 51 2021-05-12T09:29:03.176 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 37 '!=' -1 2021-05-12T09:29:03.176 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T09:29:03.177 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=37 2021-05-12T09:29:03.177 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:29:03.177 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:29:03.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:29:03.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:29:03.274 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:29:03.274 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:29:03.274 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:29:03.275 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:29:03.275 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:29:03.275 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:29:03.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=37 2021-05-12T09:29:03.648 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:29:03.648 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:29:03.648 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:29:04.142 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 37 = 51 2021-05-12T09:29:04.142 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 37 '!=' 37 2021-05-12T09:29:04.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:29:04.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:29:04.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:29:04.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:29:04.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:29:04.144 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:29:04.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:29:04.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=null 2021-05-12T09:29:04.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test null '!=' null 2021-05-12T09:29:04.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: (( 1 >= 13 )) 2021-05-12T09:29:04.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:29:04.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.2 2021-05-12T09:29:04.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:29:04.845 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:29:04.846 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:29:04.846 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:29:04.846 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:29:04.846 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:29:04.847 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:29:04.847 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:29:05.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=51 2021-05-12T09:29:05.224 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:29:05.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:29:05.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:29:05.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 51 = 51 2021-05-12T09:29:05.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:29:05.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:29:05.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:32: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.0 2021-05-12T09:29:05.767 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:29:05.767 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:29:05.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:29:05.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:29:05.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:29:05.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:29:05.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:33: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.1 2021-05-12T09:29:05.873 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:29:05.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:29:05.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:29:05.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:29:05.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:29:05.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:29:05.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:34: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.2 2021-05-12T09:29:05.979 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:29:05.979 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:29:05.980 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:29:05.980 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:29:05.980 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:29:06.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:29:06.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:35: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/0 --op remove --pgid 1.0 --force 2021-05-12T09:29:07.219 INFO:tasks.workunit.client.0.smithi028.stderr:PG '1.0' not found 2021-05-12T09:29:07.488 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:36: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/1 --op remove --pgid 1.0 --force 2021-05-12T09:29:08.847 INFO:tasks.workunit.client.0.smithi028.stdout: marking collection for removal 2021-05-12T09:29:08.847 INFO:tasks.workunit.client.0.smithi028.stdout:setting '_remove' omap key 2021-05-12T09:29:08.848 INFO:tasks.workunit.client.0.smithi028.stdout:finish_remove_pgs 1.0_head removing 1.0 2021-05-12T09:29:08.955 INFO:tasks.workunit.client.0.smithi028.stdout:Remove successful 2021-05-12T09:29:09.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:37: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/2 --op remove --pgid 1.0 --force 2021-05-12T09:29:10.433 INFO:tasks.workunit.client.0.smithi028.stderr:PG '1.0' not found 2021-05-12T09:29:10.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:38: TEST_reuse_id: activate_osd td/osd-force-create-pg 0 2021-05-12T09:29:10.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-force-create-pg 2021-05-12T09:29:10.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:29:10.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T09:29:10.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:29:10.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-force-create-pg/0 2021-05-12T09:29:10.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T09:29:10.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:29:10.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:29:10.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:29:10.715 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2021-05-12T09:29:10.715 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2021-05-12T09:29:10.715 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:29:10.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:29:10.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T09:29:10.716 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:29:10.717 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:29:10.717 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:29:10.717 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:29:10.718 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:29:10.718 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156168 2021-05-12T09:29:10.719 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156168/$cluster-$name.asok' 2021-05-12T09:29:10.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' 2021-05-12T09:29:10.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:29:10.719 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T09:29:10.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T09:29:10.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:29:10.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:29:10.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:29:10.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:29:10.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:29:10.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-force-create-pg/0 2021-05-12T09:29:10.722 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:29:10.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T09:29:10.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:29:10.724 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-force-create-pg/0/whoami 2021-05-12T09:29:10.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T09:29:10.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:29:10.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:29:10.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:29:10.731 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:10.730+0000 7f588c0df0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:10.734 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:10.732+0000 7f588c0df0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:10.736 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:10.734+0000 7f588c0df0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:11.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T09:29:11.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:29:11.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:29:11.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:29:11.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:29:11.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:11.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:29:11.100 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:29:11.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:11.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:29:11.261 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:11.259+0000 7f588c0df0c0 -1 Falling back to public interface 2021-05-12T09:29:11.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:29:12.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:29:12.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:12.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:29:12.498 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:29:12.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:12.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:29:12.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:29:13.628 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:13.626+0000 7f588c0df0c0 -1 osd.0 23 log_to_monitors {default=true} 2021-05-12T09:29:13.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:29:13.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:13.889 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:29:13.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:29:13.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:13.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:29:14.304 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 27 up_thru 18 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/162356,v1:127.0.0.1:6803/162356] [v2:127.0.0.1:6804/162356,v1:127.0.0.1:6805/162356] exists,up 9592a29b-2fe3-4bbc-ace2-fd9908c5559c 2021-05-12T09:29:14.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:29:14.309 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:29:14.309 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:29:14.309 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:39: TEST_reuse_id: activate_osd td/osd-force-create-pg 1 2021-05-12T09:29:14.309 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-force-create-pg 2021-05-12T09:29:14.310 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:29:14.310 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=1 2021-05-12T09:29:14.310 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:29:14.310 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-force-create-pg/1 2021-05-12T09:29:14.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T09:29:14.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:29:14.312 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:29:14.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:29:14.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2021-05-12T09:29:14.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2021-05-12T09:29:14.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:29:14.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:29:14.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T09:29:14.314 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:29:14.314 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:29:14.315 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:29:14.315 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:29:14.315 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:29:14.315 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156168 2021-05-12T09:29:14.316 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156168/$cluster-$name.asok' 2021-05-12T09:29:14.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' 2021-05-12T09:29:14.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:29:14.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T09:29:14.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T09:29:14.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:29:14.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:29:14.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:29:14.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:29:14.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:29:14.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-force-create-pg/1 2021-05-12T09:29:14.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.1 2021-05-12T09:29:14.319 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:29:14.320 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 1 --fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:29:14.320 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-force-create-pg/1/whoami 2021-05-12T09:29:14.320 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 1 = 1 ']' 2021-05-12T09:29:14.320 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:29:14.321 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:29:14.321 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:29:14.326 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:14.324+0000 7f8bd86ad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:14.372 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:14.371+0000 7f8bd86ad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:14.374 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:14.372+0000 7f8bd86ad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:14.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 1 2021-05-12T09:29:14.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:29:14.697 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:29:14.697 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:29:14.697 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:29:14.697 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:14.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:29:14.698 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:29:14.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:14.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:29:14.890 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:14.888+0000 7f8bd86ad0c0 -1 Falling back to public interface 2021-05-12T09:29:15.075 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:29:16.076 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:29:16.076 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:16.076 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:29:16.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:29:16.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:16.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:29:16.464 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:29:17.202 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:17.199+0000 7f8bd86ad0c0 -1 osd.1 23 log_to_monitors {default=true} 2021-05-12T09:29:17.465 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:29:17.465 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:17.466 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:29:17.466 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:29:17.467 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:17.467 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:29:17.863 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 30 up_thru 17 down_at 25 last_clean_interval [10,23) [v2:127.0.0.1:6810/162913,v1:127.0.0.1:6811/162913] [v2:127.0.0.1:6812/162913,v1:127.0.0.1:6813/162913] exists,up 657ba8d3-e8fb-467b-a80b-a3785bc02451 2021-05-12T09:29:17.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:29:17.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:29:17.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:29:17.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:40: TEST_reuse_id: activate_osd td/osd-force-create-pg 2 2021-05-12T09:29:17.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-force-create-pg 2021-05-12T09:29:17.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:29:17.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=2 2021-05-12T09:29:17.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:29:17.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-force-create-pg/2 2021-05-12T09:29:17.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-05-12T09:29:17.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:29:17.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:29:17.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:29:17.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2021-05-12T09:29:17.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2021-05-12T09:29:17.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:29:17.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:29:17.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-05-12T09:29:17.871 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:29:17.872 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:29:17.872 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:29:17.872 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:29:17.872 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:29:17.873 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156168 2021-05-12T09:29:17.873 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.156168/$cluster-$name.asok' 2021-05-12T09:29:17.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' 2021-05-12T09:29:17.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:29:17.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-05-12T09:29:17.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-05-12T09:29:17.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:29:17.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:29:17.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:29:17.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:29:17.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:29:17.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-force-create-pg/2 2021-05-12T09:29:17.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.2 2021-05-12T09:29:17.876 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:29:17.877 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 2 --fsid=72ad179f-41ed-4f43-ab5a-5f8bf1b1dbfb --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.156168/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:29:17.877 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-force-create-pg/2/whoami 2021-05-12T09:29:17.877 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 2 = 2 ']' 2021-05-12T09:29:17.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:29:17.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:29:17.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:29:17.883 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:17.881+0000 7feddbddc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:17.905 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:17.903+0000 7feddbddc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:17.907 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:17.905+0000 7feddbddc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:18.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 2 2021-05-12T09:29:18.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:29:18.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:29:18.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:29:18.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:29:18.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:18.293 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:29:18.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:29:18.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:18.295 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:29:18.427 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:18.424+0000 7feddbddc0c0 -1 Falling back to public interface 2021-05-12T09:29:18.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:29:19.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:29:19.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:19.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:29:19.694 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:29:19.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:19.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:29:20.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:29:20.668 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:20.666+0000 7feddbddc0c0 -1 osd.2 23 log_to_monitors {default=true} 2021-05-12T09:29:21.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:29:21.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:21.091 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:29:21.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:29:21.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:21.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:29:21.499 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 33 up_thru 33 down_at 28 last_clean_interval [15,23) [v2:127.0.0.1:6818/163470,v1:127.0.0.1:6819/163470] [v2:127.0.0.1:6820/163470,v1:127.0.0.1:6821/163470] exists,up 7e8a2c8c-5f59-4812-8149-741863d734d5 2021-05-12T09:29:21.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:29:21.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:29:21.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:29:21.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:41: TEST_reuse_id: sleep 10 2021-05-12T09:29:31.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: ceph pg ls 2021-05-12T09:29:31.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: grep 1.0 2021-05-12T09:29:31.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: grep stale 2021-05-12T09:29:31.877 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 0 0 0 0 0 0 0 0 stale+active+clean 33s 0'0 21:11 [1]p1 [1]p1 2021-05-12T09:28:57.755065+0000 2021-05-12T09:28:57.755065+0000 2021-05-12T09:29:31.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:44: TEST_reuse_id: ceph osd force-create-pg 1.0 --yes-i-really-mean-it 2021-05-12T09:29:32.396 INFO:tasks.workunit.client.0.smithi028.stderr:pg 1.0 already creating 2021-05-12T09:29:32.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:45: TEST_reuse_id: wait_for_clean 2021-05-12T09:29:32.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:29:32.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:29:32.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:29:32.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:29:32.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T09:29:32.408 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:29:32.408 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:29:32.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:29:32.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:29:32.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:29:32.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:29:32.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:29:32.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:29:32.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:29:32.497 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:29:32.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:29:32.896 INFO:tasks.workunit.client.0.smithi028.stderr:1 2021-05-12T09:29:32.896 INFO:tasks.workunit.client.0.smithi028.stderr:2' 2021-05-12T09:29:32.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:29:32.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:29:32.898 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:29:33.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=115964116998 2021-05-12T09:29:33.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 115964116998 2021-05-12T09:29:33.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-115964116998' 2021-05-12T09:29:33.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:29:33.026 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:29:33.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=128849018886 2021-05-12T09:29:33.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 128849018886 2021-05-12T09:29:33.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-115964116998 1-128849018886' 2021-05-12T09:29:33.158 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:29:33.159 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-05-12T09:29:33.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=141733920773 2021-05-12T09:29:33.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 141733920773 2021-05-12T09:29:33.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-115964116998 1-128849018886 2-141733920773' 2021-05-12T09:29:33.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:29:33.294 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-115964116998 2021-05-12T09:29:33.294 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:29:33.295 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:29:33.295 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-115964116998 2021-05-12T09:29:33.295 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:29:33.295 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=115964116998 2021-05-12T09:29:33.295 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 115964116998' 2021-05-12T09:29:33.296 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 115964116998 2021-05-12T09:29:33.296 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:29:33.660 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 115964116996 -lt 115964116998 2021-05-12T09:29:33.660 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:29:34.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:29:34.664 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:29:35.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 115964116999 -lt 115964116998 2021-05-12T09:29:35.049 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:29:35.050 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-128849018886 2021-05-12T09:29:35.050 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:29:35.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:29:35.052 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-128849018886 2021-05-12T09:29:35.052 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:29:35.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=128849018886 2021-05-12T09:29:35.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 128849018886' 2021-05-12T09:29:35.053 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 128849018886 2021-05-12T09:29:35.054 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:29:35.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 128849018886 -lt 128849018886 2021-05-12T09:29:35.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:29:35.536 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-141733920773 2021-05-12T09:29:35.536 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:29:35.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=2 2021-05-12T09:29:35.537 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 2-141733920773 2021-05-12T09:29:35.538 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:29:35.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=141733920773 2021-05-12T09:29:35.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.2 seq 141733920773' 2021-05-12T09:29:35.539 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.2 seq 141733920773 2021-05-12T09:29:35.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 2 2021-05-12T09:29:35.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 141733920773 -lt 141733920773 2021-05-12T09:29:35.915 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:29:35.915 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:29:35.915 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:29:36.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 51 == 0 2021-05-12T09:29:36.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:29:36.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:29:36.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:29:36.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:29:36.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:29:36.410 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:29:36.410 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:29:36.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=51 2021-05-12T09:29:36.781 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:29:36.781 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:29:36.781 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:29:37.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 51 = 51 2021-05-12T09:29:37.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:29:37.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:29:37.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T09:29:37.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-force-create-pg 0 2021-05-12T09:29:37.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2021-05-12T09:29:37.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T09:29:37.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2021-05-12T09:29:37.278 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:29:37.278 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:29:37.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:29:37.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:29:37.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:29:37.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:29:37.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:29:37.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:29:37.385 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:29:37.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:29:37.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:29:37.387 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:29:37.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:29:37.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:29:37.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:29:37.389 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:29:37.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:29:37.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T09:29:37.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2021-05-12T09:29:37.397 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:29:37.397 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:29:37.398 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.156168 2021-05-12T09:29:37.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.156168 2021-05-12T09:29:37.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:29:37.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:29:37.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T09:29:37.401 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T09:29:37.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T09:29:37.466 INFO:tasks.workunit:Running workunit osd/osd-markdown.sh... 2021-05-12T09:29:37.466 DEBUG:teuthology.orchestra.run.smithi028:workunit test osd/osd-markdown.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2021-05-12T09:29:37.527 INFO:tasks.workunit.client.0.smithi028.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T09:29:37.534 INFO:tasks.workunit.client.0.smithi028.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T09:29:37.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:29:37.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:29:37.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T09:29:37.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T09:29:37.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T09:29:37.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T09:29:37.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T09:29:37.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T09:29:37.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-markdown 2021-05-12T09:29:37.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:22: run: local dir=td/osd-markdown 2021-05-12T09:29:37.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:23: run: shift 2021-05-12T09:29:37.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: export CEPH_MON=127.0.0.1:7108 2021-05-12T09:29:37.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: CEPH_MON=127.0.0.1:7108 2021-05-12T09:29:37.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:26: run: export CEPH_ARGS 2021-05-12T09:29:37.538 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: uuidgen 2021-05-12T09:29:37.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: CEPH_ARGS+='--fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none ' 2021-05-12T09:29:37.538 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7108 ' 2021-05-12T09:29:37.540 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: set 2021-05-12T09:29:37.540 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T09:29:37.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: local 'funcs=TEST_markdown_boot 2021-05-12T09:29:37.544 INFO:tasks.workunit.client.0.smithi028.stderr:TEST_markdown_boot_exceed_time 2021-05-12T09:29:37.544 INFO:tasks.workunit.client.0.smithi028.stderr:TEST_markdown_exceed_maxdown_count 2021-05-12T09:29:37.544 INFO:tasks.workunit.client.0.smithi028.stderr:TEST_osd_stop' 2021-05-12T09:29:37.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-05-12T09:29:37.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-05-12T09:29:37.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-05-12T09:29:37.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-05-12T09:29:37.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T09:29:37.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:29:37.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T09:29:37.546 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:29:37.546 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:29:37.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:29:37.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:29:37.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:29:37.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:29:37.547 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:29:37.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:29:37.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:29:37.549 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:29:37.550 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:29:37.550 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:29:37.550 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:29:37.550 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:29:37.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:29:37.551 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:29:37.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:29:37.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:29:37.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T09:29:37.555 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:29:37.555 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:29:37.556 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:29:37.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164527 2021-05-12T09:29:37.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:29:37.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:29:37.557 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-05-12T09:29:37.557 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:29:37.557 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:29:37.557 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:29:37.557 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.164527 2021-05-12T09:29:37.558 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:29:37.558 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T09:29:37.558 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T09:29:37.558 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:29:37.559 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-05-12T09:29:37.559 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot td/osd-markdown 2021-05-12T09:29:37.559 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:82: TEST_markdown_boot: local dir=td/osd-markdown 2021-05-12T09:29:37.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:84: TEST_markdown_boot: run_mon td/osd-markdown a 2021-05-12T09:29:37.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-05-12T09:29:37.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:29:37.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:29:37.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:29:37.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-05-12T09:29:37.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-05-12T09:29:37.901 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:29:37.901 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:29:37.902 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:29:37.902 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:29:37.902 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:29:37.902 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:29:37.903 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:29:37.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:29:38.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:29:38.156 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:29:38.158 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:29:38.158 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:29:38.159 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:29:38.159 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:29:38.159 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:29:38.159 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:29:38.160 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:29:38.160 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:29:38.161 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:29:38.161 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:29:38.161 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164527/ceph-mon.a.asok 2021-05-12T09:29:38.161 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:29:38.161 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164527/ceph-mon.a.asok config get fsid 2021-05-12T09:29:38.265 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:29:38.265 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:29:38.266 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:29:38.266 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:29:38.266 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:29:38.267 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:29:38.267 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:29:38.267 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:29:38.267 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:29:38.268 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:29:38.268 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:29:38.268 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164527/ceph-mon.a.asok 2021-05-12T09:29:38.268 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:29:38.269 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164527/ceph-mon.a.asok config get mon_host 2021-05-12T09:29:38.353 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:85: TEST_markdown_boot: run_mgr td/osd-markdown x 2021-05-12T09:29:38.353 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-05-12T09:29:38.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:29:38.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:29:38.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:29:38.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-05-12T09:29:38.355 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:29:39.235 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:29:39.236 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:29:39.236 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:29:39.236 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:29:39.236 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:29:39.237 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:29:39.237 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:29:39.237 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:29:39.238 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:29:39.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:86: TEST_markdown_boot: run_osd td/osd-markdown 0 2021-05-12T09:29:39.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T09:29:39.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:29:39.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:29:39.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:29:39.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-05-12T09:29:39.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T09:29:39.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:29:39.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:29:39.265 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:29:39.265 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-05-12T09:29:39.265 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-05-12T09:29:39.265 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:29:39.266 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:29:39.266 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T09:29:39.267 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:29:39.267 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:29:39.267 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:29:39.267 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:29:39.268 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:29:39.268 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:29:39.268 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:29:39.268 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:29:39.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:29:39.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:29:39.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:29:39.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T09:29:39.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T09:29:39.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:29:39.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:29:39.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:29:39.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:29:39.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:29:39.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/0 2021-05-12T09:29:39.271 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:29:39.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=ec546ff1-6e7e-4859-8f0d-fe11f8a38fea 2021-05-12T09:29:39.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 ec546ff1-6e7e-4859-8f0d-fe11f8a38fea' 2021-05-12T09:29:39.272 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 ec546ff1-6e7e-4859-8f0d-fe11f8a38fea 2021-05-12T09:29:39.272 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:29:39.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQADoJtg5SZFEBAA4WnWHrJj18sg1UkvIL3Kkw== 2021-05-12T09:29:39.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQADoJtg5SZFEBAA4WnWHrJj18sg1UkvIL3Kkw=="}' 2021-05-12T09:29:39.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new ec546ff1-6e7e-4859-8f0d-fe11f8a38fea -i td/osd-markdown/0/new.json 2021-05-12T09:29:39.604 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:29:39.614 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/0/new.json 2021-05-12T09:29:39.615 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQADoJtg5SZFEBAA4WnWHrJj18sg1UkvIL3Kkw== --osd-uuid ec546ff1-6e7e-4859-8f0d-fe11f8a38fea 2021-05-12T09:29:39.637 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:39.634+0000 7f39ac2ee0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:39.639 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:39.636+0000 7f39ac2ee0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:39.640 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:39.637+0000 7f39ac2ee0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:39.641 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:39.638+0000 7f39ac2ee0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T09:29:39.641 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:39.638+0000 7f39ac2ee0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T09:29:39.641 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:39.638+0000 7f39ac2ee0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T09:29:39.676 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:39.673+0000 7f39ac2ee0c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-05-12T09:29:44.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-05-12T09:29:44.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:29:44.312 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:29:44.312 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:29:44.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:29:44.817 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:29:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:29:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:29:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:29:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:29:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:29:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:44.835+0000 7fab11ded0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:44.837+0000 7fab11ded0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:44.838+0000 7fab11ded0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:45.197 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:29:45.197 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:29:45.198 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:29:45.198 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:29:45.198 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:29:45.198 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:45.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:29:45.199 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:29:45.200 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:45.200 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:29:45.362 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:45.358+0000 7fab11ded0c0 -1 Falling back to public interface 2021-05-12T09:29:45.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:29:46.591 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:29:46.592 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:46.592 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:29:46.593 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:29:46.593 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:46.593 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:29:46.981 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:29:47.603 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:47.601+0000 7fab11ded0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:29:47.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:29:47.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:47.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:29:47.983 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:29:47.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:47.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:29:48.401 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/165647,v1:127.0.0.1:6803/165647] [v2:127.0.0.1:6804/165647,v1:127.0.0.1:6805/165647] exists,up ec546ff1-6e7e-4859-8f0d-fe11f8a38fea 2021-05-12T09:29:48.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:29:48.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:29:48.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:29:48.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:87: TEST_markdown_boot: run_osd td/osd-markdown 1 2021-05-12T09:29:48.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T09:29:48.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:29:48.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:29:48.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:29:48.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-05-12T09:29:48.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T09:29:48.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:29:48.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:29:48.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:29:48.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-05-12T09:29:48.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-05-12T09:29:48.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:29:48.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:29:48.410 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T09:29:48.410 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:29:48.410 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:29:48.410 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:29:48.411 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:29:48.411 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:29:48.411 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:29:48.411 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:29:48.411 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:29:48.412 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:29:48.412 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:29:48.412 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:29:48.412 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T09:29:48.413 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T09:29:48.413 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:29:48.413 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:29:48.413 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:29:48.413 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:29:48.414 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:29:48.414 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/1 2021-05-12T09:29:48.414 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:29:48.415 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=baad3ce9-26f5-4cc1-ba7b-5603a66cd957 2021-05-12T09:29:48.415 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 baad3ce9-26f5-4cc1-ba7b-5603a66cd957' 2021-05-12T09:29:48.416 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 baad3ce9-26f5-4cc1-ba7b-5603a66cd957 2021-05-12T09:29:48.416 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:29:48.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAMoJtg64LLGBAArrWvLe3JB3U525xcEcPMpg== 2021-05-12T09:29:48.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAMoJtg64LLGBAArrWvLe3JB3U525xcEcPMpg=="}' 2021-05-12T09:29:48.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new baad3ce9-26f5-4cc1-ba7b-5603a66cd957 -i td/osd-markdown/1/new.json 2021-05-12T09:29:48.939 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:29:48.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/1/new.json 2021-05-12T09:29:48.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAMoJtg64LLGBAArrWvLe3JB3U525xcEcPMpg== --osd-uuid baad3ce9-26f5-4cc1-ba7b-5603a66cd957 2021-05-12T09:29:48.974 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:48.971+0000 7f356ca2c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:48.975 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:48.973+0000 7f356ca2c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:48.976 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:48.975+0000 7f356ca2c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:48.977 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:48.975+0000 7f356ca2c0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T09:29:48.977 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:48.975+0000 7f356ca2c0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T09:29:48.978 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:48.975+0000 7f356ca2c0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T09:29:49.073 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:49.071+0000 7f356ca2c0c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-05-12T09:29:53.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-05-12T09:29:53.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:29:53.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:29:53.709 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:29:53.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:29:54.223 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:29:54.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:29:54.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:29:54.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:29:54.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:29:54.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:29:54.243 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:54.241+0000 7f4ca2a050c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:54.245 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:54.243+0000 7f4ca2a050c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:54.246 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:54.244+0000 7f4ca2a050c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:54.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:29:54.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:29:54.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:29:54.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:29:54.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:29:54.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:54.611 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:29:54.612 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:29:54.612 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:54.612 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:29:54.763 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:54.762+0000 7f4ca2a050c0 -1 Falling back to public interface 2021-05-12T09:29:54.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:29:56.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:29:56.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:56.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:29:56.001 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:29:56.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:56.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:29:56.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:29:56.918 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:56.916+0000 7f4ca2a050c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:29:57.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:29:57.392 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:29:57.392 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:29:57.396 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:29:57.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:29:57.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:29:57.797 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/167032,v1:127.0.0.1:6811/167032] [v2:127.0.0.1:6812/167032,v1:127.0.0.1:6813/167032] exists,up baad3ce9-26f5-4cc1-ba7b-5603a66cd957 2021-05-12T09:29:57.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:29:57.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:29:57.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:29:57.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:88: TEST_markdown_boot: run_osd td/osd-markdown 2 2021-05-12T09:29:57.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T09:29:57.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:29:57.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T09:29:57.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:29:57.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-05-12T09:29:57.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T09:29:57.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:29:57.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:29:57.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:29:57.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-05-12T09:29:57.804 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-05-12T09:29:57.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:29:57.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:29:57.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T09:29:57.805 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:29:57.806 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:29:57.806 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:29:57.806 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:29:57.807 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:29:57.807 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:29:57.807 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:29:57.807 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:29:57.807 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:29:57.808 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:29:57.808 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:29:57.808 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T09:29:57.808 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T09:29:57.808 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:29:57.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:29:57.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:29:57.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:29:57.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:29:57.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/2 2021-05-12T09:29:57.810 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:29:57.811 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=1155bb8e-39ff-47c7-a716-e20695c6e137 2021-05-12T09:29:57.811 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 1155bb8e-39ff-47c7-a716-e20695c6e137' 2021-05-12T09:29:57.811 INFO:tasks.workunit.client.0.smithi028.stdout:add osd2 1155bb8e-39ff-47c7-a716-e20695c6e137 2021-05-12T09:29:57.812 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:29:57.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAVoJtgfl9/MBAA0ssjjahtUL/lC6Cz017SvA== 2021-05-12T09:29:57.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAVoJtgfl9/MBAA0ssjjahtUL/lC6Cz017SvA=="}' 2021-05-12T09:29:57.816 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 1155bb8e-39ff-47c7-a716-e20695c6e137 -i td/osd-markdown/2/new.json 2021-05-12T09:29:58.396 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:29:58.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/2/new.json 2021-05-12T09:29:58.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAVoJtgfl9/MBAA0ssjjahtUL/lC6Cz017SvA== --osd-uuid 1155bb8e-39ff-47c7-a716-e20695c6e137 2021-05-12T09:29:58.430 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:58.428+0000 7f7f481f90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:58.432 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:58.430+0000 7f7f481f90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:58.433 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:58.431+0000 7f7f481f90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:29:58.434 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:58.432+0000 7f7f481f90c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T09:29:58.434 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:58.432+0000 7f7f481f90c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T09:29:58.434 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:58.432+0000 7f7f481f90c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T09:29:58.471 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:29:58.469+0000 7f7f481f90c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-05-12T09:30:03.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-05-12T09:30:03.374 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:30:03.374 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T09:30:03.375 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd2 key to auth repository 2021-05-12T09:30:03.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:30:03.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T09:30:03.888 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:30:03.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:30:03.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:30:03.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:30:03.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:30:03.907 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:30:03.903+0000 7f16ec0e30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:30:03.908 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:30:03.905+0000 7f16ec0e30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:30:03.910 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:30:03.907+0000 7f16ec0e30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:30:04.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T09:30:04.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:30:04.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:30:04.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:30:04.284 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:30:04.285 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:30:04.285 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:30:04.286 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:30:04.286 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:30:04.286 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:30:04.429 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:30:04.427+0000 7f16ec0e30c0 -1 Falling back to public interface 2021-05-12T09:30:04.659 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:30:05.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:30:05.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:30:05.662 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:30:05.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:30:05.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:30:05.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:30:06.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:30:06.665 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:30:06.662+0000 7f16ec0e30c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T09:30:07.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:30:07.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:30:07.049 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:30:07.049 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:30:07.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:30:07.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:30:07.461 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/168425,v1:127.0.0.1:6819/168425] [v2:127.0.0.1:6820/168425,v1:127.0.0.1:6821/168425] exists,up 1155bb8e-39ff-47c7-a716-e20695c6e137 2021-05-12T09:30:07.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:30:07.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:30:07.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:30:07.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:90: TEST_markdown_boot: create_rbd_pool 2021-05-12T09:30:07.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T09:30:07.905 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'rbd' does not exist 2021-05-12T09:30:07.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T09:30:07.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T09:30:08.755 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'rbd' already exists 2021-05-12T09:30:08.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:30:09.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T09:30:11.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:93: TEST_markdown_boot: local count=3 2021-05-12T09:30:11.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:94: TEST_markdown_boot: local sleeptime=10 2021-05-12T09:30:11.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:95: TEST_markdown_boot: local period=120 2021-05-12T09:30:11.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:96: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-05-12T09:30:11.966 INFO:tasks.workunit.client.0.smithi028.stdout:{} 2021-05-12T09:30:11.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:97: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_period 120' 2021-05-12T09:30:12.100 INFO:tasks.workunit.client.0.smithi028.stdout:{} 2021-05-12T09:30:12.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:99: TEST_markdown_boot: markdown_N_impl 3 120 10 2021-05-12T09:30:12.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=3 2021-05-12T09:30:12.111 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=120 2021-05-12T09:30:12.111 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-05-12T09:30:12.112 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 3 2021-05-12T09:30:12.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T09:30:12.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T09:30:12.243 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T09:30:12.649 INFO:tasks.workunit.client.0.smithi028.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T09:30:12.649 INFO:tasks.workunit.client.0.smithi028.stdout:-1 0.29306 root default 2021-05-12T09:30:12.650 INFO:tasks.workunit.client.0.smithi028.stdout:-3 0.29306 host smithi028 2021-05-12T09:30:12.650 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:30:12.651 INFO:tasks.workunit.client.0.smithi028.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T09:30:12.651 INFO:tasks.workunit.client.0.smithi028.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T09:30:12.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T09:30:12.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T09:30:12.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T09:30:13.064 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:30:13.065 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T09:30:13.065 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T09:30:13.652 INFO:tasks.workunit.client.0.smithi028.stderr:marked down osd.0. 2021-05-12T09:30:13.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T09:30:23.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T09:30:23.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T09:30:23.807 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T09:30:24.197 INFO:tasks.workunit.client.0.smithi028.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T09:30:24.197 INFO:tasks.workunit.client.0.smithi028.stdout:-1 0.29306 root default 2021-05-12T09:30:24.197 INFO:tasks.workunit.client.0.smithi028.stdout:-3 0.29306 host smithi028 2021-05-12T09:30:24.197 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:30:24.198 INFO:tasks.workunit.client.0.smithi028.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T09:30:24.198 INFO:tasks.workunit.client.0.smithi028.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T09:30:24.208 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T09:30:24.209 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T09:30:24.209 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T09:30:24.583 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:30:24.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T09:30:24.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T09:30:24.976 INFO:tasks.workunit.client.0.smithi028.stderr:marked down osd.0. 2021-05-12T09:30:24.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T09:30:34.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T09:30:34.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T09:30:35.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T09:30:35.495 INFO:tasks.workunit.client.0.smithi028.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T09:30:35.496 INFO:tasks.workunit.client.0.smithi028.stdout:-1 0.29306 root default 2021-05-12T09:30:35.496 INFO:tasks.workunit.client.0.smithi028.stdout:-3 0.29306 host smithi028 2021-05-12T09:30:35.496 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:30:35.497 INFO:tasks.workunit.client.0.smithi028.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T09:30:35.497 INFO:tasks.workunit.client.0.smithi028.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T09:30:35.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T09:30:35.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T09:30:35.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T09:30:35.909 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:30:35.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T09:30:35.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T09:30:36.468 INFO:tasks.workunit.client.0.smithi028.stderr:marked down osd.0. 2021-05-12T09:30:36.477 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T09:30:46.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:101: TEST_markdown_boot: sleep 15 2021-05-12T09:31:01.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:102: TEST_markdown_boot: ceph tell osd.0 get_latest_osdmap 2021-05-12T09:31:01.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: ceph osd tree 2021-05-12T09:31:01.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep up 2021-05-12T09:31:01.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep osd.0 2021-05-12T09:31:01.999 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:31:02.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-05-12T09:31:02.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T09:31:02.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:31:02.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T09:31:02.002 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:31:02.002 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:31:02.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:31:02.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:31:02.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:31:02.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:31:02.116 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:31:02.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:31:02.116 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:31:02.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:31:02.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:31:02.117 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:31:02.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:31:02.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:31:02.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:31:02.118 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:31:02.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:31:02.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:31:02.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T09:31:02.123 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:31:02.124 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:31:02.124 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:31:02.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164527 2021-05-12T09:31:02.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:31:02.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:31:02.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-05-12T09:31:02.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-05-12T09:31:02.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-05-12T09:31:02.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-05-12T09:31:02.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T09:31:02.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:31:02.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T09:31:02.126 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:31:02.128 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:31:02.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:31:02.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:31:02.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:31:02.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:31:02.129 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:31:02.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:31:02.130 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:31:02.130 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:31:02.130 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:31:02.130 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:31:02.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:31:02.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:31:02.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:31:02.131 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:31:02.133 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:31:02.133 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:31:02.134 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T09:31:02.134 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:31:02.134 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:31:02.134 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:31:02.135 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164527 2021-05-12T09:31:02.135 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:31:02.135 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:31:02.135 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-05-12T09:31:02.136 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:31:02.136 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:31:02.136 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:31:02.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.164527 2021-05-12T09:31:02.138 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:31:02.138 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T09:31:02.139 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:31:02.139 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-05-12T09:31:02.139 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot_exceed_time td/osd-markdown 2021-05-12T09:31:02.139 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:107: TEST_markdown_boot_exceed_time: local dir=td/osd-markdown 2021-05-12T09:31:02.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:109: TEST_markdown_boot_exceed_time: run_mon td/osd-markdown a 2021-05-12T09:31:02.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-05-12T09:31:02.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:31:02.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:31:02.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:31:02.141 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-05-12T09:31:02.141 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-05-12T09:31:02.492 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:31:02.493 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:31:02.493 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:31:02.494 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:31:02.494 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:31:02.494 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:31:02.495 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:31:02.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:31:02.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:31:02.734 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:31:02.734 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:31:02.735 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:31:02.736 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:31:02.738 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:31:02.741 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:31:02.741 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:31:02.742 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:31:02.742 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:31:02.742 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:31:02.744 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:31:02.744 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164527/ceph-mon.a.asok 2021-05-12T09:31:02.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:31:02.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164527/ceph-mon.a.asok config get fsid 2021-05-12T09:31:02.830 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:31:02.830 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:31:02.830 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:31:02.831 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:31:02.831 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:31:02.831 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:31:02.831 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:31:02.831 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:31:02.832 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:31:02.832 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:31:02.832 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:31:02.832 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164527/ceph-mon.a.asok 2021-05-12T09:31:02.833 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:31:02.833 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164527/ceph-mon.a.asok config get mon_host 2021-05-12T09:31:02.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:110: TEST_markdown_boot_exceed_time: run_mgr td/osd-markdown x 2021-05-12T09:31:02.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-05-12T09:31:02.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:31:02.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:31:02.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:31:02.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-05-12T09:31:02.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:31:03.807 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:31:03.808 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:31:03.808 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:31:03.808 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:31:03.808 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:31:03.809 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:31:03.809 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:31:03.810 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:31:03.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:31:03.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:111: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 0 2021-05-12T09:31:03.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T09:31:03.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:31:03.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:31:03.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:31:03.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-05-12T09:31:03.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T09:31:03.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:31:03.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:31:03.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:31:03.839 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-05-12T09:31:03.839 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-05-12T09:31:03.839 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:31:03.839 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:31:03.840 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T09:31:03.840 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:31:03.840 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:31:03.840 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:31:03.841 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:31:03.841 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:31:03.841 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:31:03.841 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:31:03.842 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:31:03.842 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:31:03.842 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:31:03.842 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:31:03.842 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T09:31:03.843 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T09:31:03.843 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:31:03.843 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:31:03.843 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:31:03.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:31:03.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:31:03.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/0 2021-05-12T09:31:03.844 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:31:03.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=59614920-9ec3-4029-937f-68810c0f66df 2021-05-12T09:31:03.845 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 59614920-9ec3-4029-937f-68810c0f66df 2021-05-12T09:31:03.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 59614920-9ec3-4029-937f-68810c0f66df' 2021-05-12T09:31:03.846 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:31:03.847 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBXoJtg2CxhMhAAQ1JeffYtPbmfQDXUyINtPQ== 2021-05-12T09:31:03.848 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBXoJtg2CxhMhAAQ1JeffYtPbmfQDXUyINtPQ=="}' 2021-05-12T09:31:03.848 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 59614920-9ec3-4029-937f-68810c0f66df -i td/osd-markdown/0/new.json 2021-05-12T09:31:04.168 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:31:04.179 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/0/new.json 2021-05-12T09:31:04.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBXoJtg2CxhMhAAQ1JeffYtPbmfQDXUyINtPQ== --osd-uuid 59614920-9ec3-4029-937f-68810c0f66df 2021-05-12T09:31:04.202 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:04.200+0000 7f0cbfe7e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:04.203 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:04.202+0000 7f0cbfe7e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:04.206 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:04.203+0000 7f0cbfe7e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:04.206 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:04.204+0000 7f0cbfe7e0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T09:31:04.206 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:04.204+0000 7f0cbfe7e0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T09:31:04.207 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:04.204+0000 7f0cbfe7e0c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T09:31:04.249 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:04.248+0000 7f0cbfe7e0c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-05-12T09:31:08.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-05-12T09:31:08.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:31:08.887 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:31:08.887 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:31:08.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:31:09.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:31:09.393 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:31:09.394 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:31:09.394 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:31:09.394 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:31:09.395 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:31:09.413 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:09.411+0000 7f43a72160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:09.417 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:09.415+0000 7f43a72160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:09.419 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:09.417+0000 7f43a72160c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:09.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:31:09.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:31:09.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:31:09.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:31:09.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:31:09.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:31:09.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:31:09.769 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:31:09.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:31:09.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:31:09.934 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:09.932+0000 7f43a72160c0 -1 Falling back to public interface 2021-05-12T09:31:10.169 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:31:11.170 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:31:11.171 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:31:11.171 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:31:11.172 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:31:11.172 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:31:11.172 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:31:11.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:31:12.153 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:12.151+0000 7f43a72160c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:31:12.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:31:12.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:31:12.562 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:31:12.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:31:12.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:31:12.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:31:12.948 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/170593,v1:127.0.0.1:6803/170593] [v2:127.0.0.1:6804/170593,v1:127.0.0.1:6805/170593] exists,up 59614920-9ec3-4029-937f-68810c0f66df 2021-05-12T09:31:12.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:31:12.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:31:12.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:31:12.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:112: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 1 2021-05-12T09:31:12.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T09:31:12.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:31:12.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:31:12.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:31:12.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-05-12T09:31:12.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T09:31:12.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:31:12.955 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:31:12.955 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:31:12.955 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-05-12T09:31:12.955 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-05-12T09:31:12.956 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:31:12.956 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:31:12.956 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T09:31:12.956 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:31:12.956 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:31:12.957 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:31:12.957 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:31:12.957 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:31:12.957 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:31:12.958 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:31:12.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:31:12.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:31:12.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:31:12.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:31:12.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T09:31:12.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T09:31:12.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:31:12.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:31:12.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:31:12.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:31:12.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:31:12.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/1 2021-05-12T09:31:12.960 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:31:12.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=48899cc7-8433-4028-b3c5-72f6d05f8ff7 2021-05-12T09:31:12.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 48899cc7-8433-4028-b3c5-72f6d05f8ff7' 2021-05-12T09:31:12.961 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 48899cc7-8433-4028-b3c5-72f6d05f8ff7 2021-05-12T09:31:12.962 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:31:12.964 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBgoJtgXn1bORAAGh/jclSo2/II/NTv/sT0dA== 2021-05-12T09:31:12.964 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBgoJtgXn1bORAAGh/jclSo2/II/NTv/sT0dA=="}' 2021-05-12T09:31:12.964 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 48899cc7-8433-4028-b3c5-72f6d05f8ff7 -i td/osd-markdown/1/new.json 2021-05-12T09:31:13.507 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:31:13.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/1/new.json 2021-05-12T09:31:13.519 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBgoJtgXn1bORAAGh/jclSo2/II/NTv/sT0dA== --osd-uuid 48899cc7-8433-4028-b3c5-72f6d05f8ff7 2021-05-12T09:31:13.542 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:13.539+0000 7fb2690ad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:13.566 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:13.563+0000 7fb2690ad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:13.567 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:13.564+0000 7fb2690ad0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:13.568 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:13.565+0000 7fb2690ad0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T09:31:13.568 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:13.565+0000 7fb2690ad0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T09:31:13.569 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:13.565+0000 7fb2690ad0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T09:31:13.624 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:13.621+0000 7fb2690ad0c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-05-12T09:31:18.477 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-05-12T09:31:18.477 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:31:18.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:31:18.478 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:31:18.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:31:18.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:31:18.989 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:31:18.990 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:31:18.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:31:18.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:31:18.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:31:19.009 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:19.006+0000 7f95ece130c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:19.011 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:19.008+0000 7f95ece130c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:19.012 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:19.009+0000 7f95ece130c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:19.364 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:31:19.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:31:19.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:31:19.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:31:19.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:31:19.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:31:19.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:31:19.366 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:31:19.367 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:31:19.367 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:31:19.534 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:19.532+0000 7f95ece130c0 -1 Falling back to public interface 2021-05-12T09:31:19.738 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:31:20.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:31:20.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:31:20.740 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:31:20.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:31:20.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:31:20.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:31:21.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:31:21.709 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:21.706+0000 7f95ece130c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:31:22.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:31:22.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:31:22.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:31:22.132 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:31:22.132 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:31:22.133 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:31:22.518 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/171978,v1:127.0.0.1:6811/171978] [v2:127.0.0.1:6812/171978,v1:127.0.0.1:6813/171978] exists,up 48899cc7-8433-4028-b3c5-72f6d05f8ff7 2021-05-12T09:31:22.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:31:22.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:31:22.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:31:22.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:113: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 2 2021-05-12T09:31:22.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T09:31:22.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:31:22.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T09:31:22.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:31:22.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-05-12T09:31:22.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T09:31:22.527 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:31:22.527 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:31:22.527 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:31:22.528 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-05-12T09:31:22.528 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-05-12T09:31:22.528 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:31:22.528 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:31:22.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T09:31:22.529 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:31:22.529 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:31:22.529 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:31:22.529 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:31:22.530 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:31:22.530 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:31:22.530 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:31:22.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:31:22.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:31:22.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:31:22.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:31:22.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T09:31:22.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T09:31:22.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:31:22.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:31:22.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:31:22.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:31:22.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:31:22.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/2 2021-05-12T09:31:22.533 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:31:22.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=e18b433e-ca71-44f4-b181-3cb36d3fca4d 2021-05-12T09:31:22.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 e18b433e-ca71-44f4-b181-3cb36d3fca4d' 2021-05-12T09:31:22.534 INFO:tasks.workunit.client.0.smithi028.stdout:add osd2 e18b433e-ca71-44f4-b181-3cb36d3fca4d 2021-05-12T09:31:22.535 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:31:22.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBqoJtgrJXEHxAAURblfCQ2hxyY6rM8XVqBNw== 2021-05-12T09:31:22.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBqoJtgrJXEHxAAURblfCQ2hxyY6rM8XVqBNw=="}' 2021-05-12T09:31:22.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new e18b433e-ca71-44f4-b181-3cb36d3fca4d -i td/osd-markdown/2/new.json 2021-05-12T09:31:23.084 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:31:23.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/2/new.json 2021-05-12T09:31:23.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBqoJtgrJXEHxAAURblfCQ2hxyY6rM8XVqBNw== --osd-uuid e18b433e-ca71-44f4-b181-3cb36d3fca4d 2021-05-12T09:31:23.118 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:23.115+0000 7f2ae20ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:23.120 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:23.117+0000 7f2ae20ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:23.122 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:23.119+0000 7f2ae20ff0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:23.123 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:23.120+0000 7f2ae20ff0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T09:31:23.123 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:23.120+0000 7f2ae20ff0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T09:31:23.123 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:23.120+0000 7f2ae20ff0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T09:31:23.164 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:23.161+0000 7f2ae20ff0c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-05-12T09:31:28.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-05-12T09:31:28.023 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:31:28.024 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd2 key to auth repository 2021-05-12T09:31:28.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T09:31:28.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:31:28.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T09:31:28.535 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:31:28.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:31:28.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:31:28.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:31:28.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:31:28.551 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:28.548+0000 7f0d70e850c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:28.553 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:28.550+0000 7f0d70e850c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:28.555 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:28.552+0000 7f0d70e850c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:31:28.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T09:31:28.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:31:28.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:31:28.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:31:28.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:31:28.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:31:28.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:31:28.913 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:31:28.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:31:28.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:31:29.071 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:29.068+0000 7f0d70e850c0 -1 Falling back to public interface 2021-05-12T09:31:29.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:31:30.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:31:30.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:31:30.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:31:30.294 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:31:30.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:31:30.295 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:31:30.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:31:31.346 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:31:31.344+0000 7f0d70e850c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T09:31:31.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:31:31.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:31:31.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:31:31.682 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:31:31.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:31:31.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:31:32.068 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/173363,v1:127.0.0.1:6819/173363] [v2:127.0.0.1:6820/173363,v1:127.0.0.1:6821/173363] exists,up e18b433e-ca71-44f4-b181-3cb36d3fca4d 2021-05-12T09:31:32.069 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:31:32.070 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:31:32.070 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:31:32.071 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:115: TEST_markdown_boot_exceed_time: create_rbd_pool 2021-05-12T09:31:32.071 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T09:31:32.521 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'rbd' does not exist 2021-05-12T09:31:32.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T09:31:32.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T09:31:33.292 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'rbd' already exists 2021-05-12T09:31:33.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:31:34.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T09:31:36.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:118: TEST_markdown_boot_exceed_time: local count=3 2021-05-12T09:31:36.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:119: TEST_markdown_boot_exceed_time: local period=20 2021-05-12T09:31:36.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:120: TEST_markdown_boot_exceed_time: local sleeptime=10 2021-05-12T09:31:36.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:121: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-05-12T09:31:36.310 INFO:tasks.workunit.client.0.smithi028.stdout:{} 2021-05-12T09:31:36.320 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:122: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_period 20' 2021-05-12T09:31:36.443 INFO:tasks.workunit.client.0.smithi028.stdout:{} 2021-05-12T09:31:36.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:124: TEST_markdown_boot_exceed_time: markdown_N_impl 4 20 10 2021-05-12T09:31:36.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2021-05-12T09:31:36.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=20 2021-05-12T09:31:36.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-05-12T09:31:36.455 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2021-05-12T09:31:36.455 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T09:31:36.456 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T09:31:36.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T09:31:36.983 INFO:tasks.workunit.client.0.smithi028.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T09:31:36.984 INFO:tasks.workunit.client.0.smithi028.stdout:-1 0.29306 root default 2021-05-12T09:31:36.984 INFO:tasks.workunit.client.0.smithi028.stdout:-3 0.29306 host smithi028 2021-05-12T09:31:36.984 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:31:36.984 INFO:tasks.workunit.client.0.smithi028.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T09:31:36.985 INFO:tasks.workunit.client.0.smithi028.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T09:31:36.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T09:31:37.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T09:31:37.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T09:31:37.371 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:31:37.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T09:31:37.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T09:31:37.874 INFO:tasks.workunit.client.0.smithi028.stderr:marked down osd.0. 2021-05-12T09:31:37.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T09:31:47.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T09:31:47.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T09:31:48.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T09:31:48.490 INFO:tasks.workunit.client.0.smithi028.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T09:31:48.490 INFO:tasks.workunit.client.0.smithi028.stdout:-1 0.29306 root default 2021-05-12T09:31:48.491 INFO:tasks.workunit.client.0.smithi028.stdout:-3 0.29306 host smithi028 2021-05-12T09:31:48.491 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:31:48.491 INFO:tasks.workunit.client.0.smithi028.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T09:31:48.491 INFO:tasks.workunit.client.0.smithi028.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T09:31:48.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T09:31:48.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T09:31:48.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T09:31:48.879 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:31:48.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T09:31:48.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T09:31:49.287 INFO:tasks.workunit.client.0.smithi028.stderr:marked down osd.0. 2021-05-12T09:31:49.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T09:31:59.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T09:31:59.298 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T09:31:59.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T09:31:59.805 INFO:tasks.workunit.client.0.smithi028.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T09:31:59.805 INFO:tasks.workunit.client.0.smithi028.stdout:-1 0.29306 root default 2021-05-12T09:31:59.806 INFO:tasks.workunit.client.0.smithi028.stdout:-3 0.29306 host smithi028 2021-05-12T09:31:59.806 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:31:59.806 INFO:tasks.workunit.client.0.smithi028.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T09:31:59.807 INFO:tasks.workunit.client.0.smithi028.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T09:31:59.817 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T09:31:59.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T09:31:59.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T09:32:00.217 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:32:00.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T09:32:00.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T09:32:00.655 INFO:tasks.workunit.client.0.smithi028.stderr:marked down osd.0. 2021-05-12T09:32:00.664 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T09:32:10.665 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T09:32:10.666 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T09:32:10.808 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T09:32:11.181 INFO:tasks.workunit.client.0.smithi028.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T09:32:11.181 INFO:tasks.workunit.client.0.smithi028.stdout:-1 0.29306 root default 2021-05-12T09:32:11.182 INFO:tasks.workunit.client.0.smithi028.stdout:-3 0.29306 host smithi028 2021-05-12T09:32:11.182 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:32:11.182 INFO:tasks.workunit.client.0.smithi028.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T09:32:11.182 INFO:tasks.workunit.client.0.smithi028.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T09:32:11.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T09:32:11.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T09:32:11.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T09:32:11.584 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:32:11.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T09:32:11.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T09:32:12.018 INFO:tasks.workunit.client.0.smithi028.stderr:marked down osd.0. 2021-05-12T09:32:12.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T09:32:22.029 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:125: TEST_markdown_boot_exceed_time: sleep 15 2021-05-12T09:32:37.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:126: TEST_markdown_boot_exceed_time: ceph tell osd.0 get_latest_osdmap 2021-05-12T09:32:37.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: ceph osd tree 2021-05-12T09:32:37.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep up 2021-05-12T09:32:37.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep osd.0 2021-05-12T09:32:37.549 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:32:37.550 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-05-12T09:32:37.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T09:32:37.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:32:37.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T09:32:37.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:32:37.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:32:37.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:32:37.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:32:37.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:32:37.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:32:37.657 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:32:37.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:32:37.658 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:32:37.659 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:32:37.659 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:32:37.659 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:32:37.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:32:37.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:32:37.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:32:37.662 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:32:37.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:32:37.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:32:37.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T09:32:37.671 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:32:37.671 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:32:37.672 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:32:37.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164527 2021-05-12T09:32:37.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:32:37.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:32:37.676 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-05-12T09:32:37.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-05-12T09:32:37.677 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-05-12T09:32:37.678 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-05-12T09:32:37.678 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T09:32:37.678 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:32:37.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T09:32:37.679 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:32:37.680 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:32:37.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:32:37.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:32:37.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:32:37.681 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:32:37.682 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:32:37.687 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:32:37.688 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:32:37.688 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:32:37.688 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:32:37.688 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:32:37.689 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:32:37.689 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:32:37.689 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:32:37.690 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:32:37.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:32:37.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:32:37.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T09:32:37.692 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:32:37.692 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:32:37.692 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:32:37.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164527 2021-05-12T09:32:37.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:32:37.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:32:37.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-05-12T09:32:37.693 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:32:37.693 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:32:37.694 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:32:37.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.164527 2021-05-12T09:32:37.694 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:32:37.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T09:32:37.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:32:37.695 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-05-12T09:32:37.695 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_exceed_maxdown_count td/osd-markdown 2021-05-12T09:32:37.695 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:59: TEST_markdown_exceed_maxdown_count: local dir=td/osd-markdown 2021-05-12T09:32:37.695 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:61: TEST_markdown_exceed_maxdown_count: run_mon td/osd-markdown a 2021-05-12T09:32:37.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-05-12T09:32:37.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:32:37.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:32:37.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:32:37.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-05-12T09:32:37.697 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-05-12T09:32:38.074 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:32:38.074 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:32:38.075 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:32:38.075 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:32:38.075 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:32:38.075 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:32:38.076 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:32:38.076 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:32:38.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:32:38.319 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:32:38.319 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:32:38.320 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:32:38.320 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:32:38.320 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:32:38.320 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:32:38.321 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:32:38.321 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:32:38.321 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:32:38.321 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:32:38.321 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:32:38.323 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164527/ceph-mon.a.asok 2021-05-12T09:32:38.324 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:32:38.324 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164527/ceph-mon.a.asok config get fsid 2021-05-12T09:32:38.423 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:32:38.423 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:32:38.423 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:32:38.424 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:32:38.424 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:32:38.424 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:32:38.424 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:32:38.425 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:32:38.425 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:32:38.425 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:32:38.426 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:32:38.426 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164527/ceph-mon.a.asok 2021-05-12T09:32:38.426 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:32:38.426 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164527/ceph-mon.a.asok config get mon_host 2021-05-12T09:32:38.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:62: TEST_markdown_exceed_maxdown_count: run_mgr td/osd-markdown x 2021-05-12T09:32:38.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-05-12T09:32:38.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:32:38.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:32:38.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:32:38.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-05-12T09:32:38.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:32:39.399 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:32:39.399 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:32:39.399 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:32:39.400 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:32:39.400 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:32:39.400 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:32:39.400 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:32:39.400 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:32:39.401 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:32:39.425 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:63: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 0 2021-05-12T09:32:39.425 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T09:32:39.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:32:39.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:32:39.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:32:39.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-05-12T09:32:39.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T09:32:39.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:32:39.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:32:39.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:32:39.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-05-12T09:32:39.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-05-12T09:32:39.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:32:39.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:32:39.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T09:32:39.429 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:32:39.429 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:32:39.429 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:32:39.429 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:32:39.429 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:32:39.430 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:32:39.430 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:32:39.430 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:32:39.430 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:32:39.430 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:32:39.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:32:39.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T09:32:39.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T09:32:39.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:32:39.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:32:39.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:32:39.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:32:39.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:32:39.433 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/0 2021-05-12T09:32:39.433 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:32:39.433 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=4d82f0d6-dc1a-4aa4-9452-13e29c7182dd 2021-05-12T09:32:39.433 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 4d82f0d6-dc1a-4aa4-9452-13e29c7182dd 2021-05-12T09:32:39.434 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 4d82f0d6-dc1a-4aa4-9452-13e29c7182dd' 2021-05-12T09:32:39.434 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:32:39.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQC3oJtgpZwCGhAADUikNGEvHtnN9ZPVW9cKgg== 2021-05-12T09:32:39.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQC3oJtgpZwCGhAADUikNGEvHtnN9ZPVW9cKgg=="}' 2021-05-12T09:32:39.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 4d82f0d6-dc1a-4aa4-9452-13e29c7182dd -i td/osd-markdown/0/new.json 2021-05-12T09:32:39.752 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:32:39.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/0/new.json 2021-05-12T09:32:39.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC3oJtgpZwCGhAADUikNGEvHtnN9ZPVW9cKgg== --osd-uuid 4d82f0d6-dc1a-4aa4-9452-13e29c7182dd 2021-05-12T09:32:39.785 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:39.782+0000 7f46907d70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:39.787 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:39.784+0000 7f46907d70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:39.789 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:39.786+0000 7f46907d70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:39.789 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:39.787+0000 7f46907d70c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T09:32:39.790 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:39.787+0000 7f46907d70c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T09:32:39.790 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:39.787+0000 7f46907d70c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T09:32:39.832 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:39.829+0000 7f46907d70c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-05-12T09:32:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-05-12T09:32:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:32:44.498 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:32:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:32:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:32:44.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:32:44.999 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:32:45.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:32:45.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:32:45.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:32:45.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:32:45.017 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:45.016+0000 7f52647a30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:45.019 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:45.018+0000 7f52647a30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:45.021 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:45.019+0000 7f52647a30c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:45.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:32:45.374 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:32:45.374 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:32:45.374 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:32:45.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:32:45.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:32:45.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:32:45.376 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:32:45.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:32:45.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:32:45.543 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:45.541+0000 7f52647a30c0 -1 Falling back to public interface 2021-05-12T09:32:45.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:32:46.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:32:46.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:32:46.748 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:32:46.748 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:32:46.748 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:32:46.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:32:47.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:32:47.760 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:47.759+0000 7f52647a30c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:32:48.133 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:32:48.133 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:32:48.133 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:32:48.134 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:32:48.134 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:32:48.134 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:32:48.570 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/175628,v1:127.0.0.1:6803/175628] [v2:127.0.0.1:6804/175628,v1:127.0.0.1:6805/175628] exists,up 4d82f0d6-dc1a-4aa4-9452-13e29c7182dd 2021-05-12T09:32:48.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:32:48.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:32:48.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:32:48.576 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:64: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 1 2021-05-12T09:32:48.576 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T09:32:48.576 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:32:48.576 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:32:48.577 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:32:48.577 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-05-12T09:32:48.577 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T09:32:48.577 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:32:48.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:32:48.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:32:48.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-05-12T09:32:48.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-05-12T09:32:48.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:32:48.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:32:48.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T09:32:48.579 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:32:48.580 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:32:48.580 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:32:48.580 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:32:48.580 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:32:48.580 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:32:48.581 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:32:48.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:32:48.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:32:48.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:32:48.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:32:48.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T09:32:48.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T09:32:48.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:32:48.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:32:48.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:32:48.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:32:48.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:32:48.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/1 2021-05-12T09:32:48.584 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:32:48.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=a2cad106-a33f-488b-9db6-e9a2fe7865cc 2021-05-12T09:32:48.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 a2cad106-a33f-488b-9db6-e9a2fe7865cc' 2021-05-12T09:32:48.584 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 a2cad106-a33f-488b-9db6-e9a2fe7865cc 2021-05-12T09:32:48.585 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:32:48.586 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDAoJtgd4vUIhAABhOG24aBqHLan3jXU9M4sw== 2021-05-12T09:32:48.586 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDAoJtgd4vUIhAABhOG24aBqHLan3jXU9M4sw=="}' 2021-05-12T09:32:48.586 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new a2cad106-a33f-488b-9db6-e9a2fe7865cc -i td/osd-markdown/1/new.json 2021-05-12T09:32:49.141 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:32:49.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/1/new.json 2021-05-12T09:32:49.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDAoJtgd4vUIhAABhOG24aBqHLan3jXU9M4sw== --osd-uuid a2cad106-a33f-488b-9db6-e9a2fe7865cc 2021-05-12T09:32:49.175 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:49.173+0000 7faa9107e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:49.176 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:49.175+0000 7faa9107e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:49.178 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:49.176+0000 7faa9107e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:49.180 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:49.176+0000 7faa9107e0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T09:32:49.180 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:49.176+0000 7faa9107e0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T09:32:49.181 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:49.176+0000 7faa9107e0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T09:32:49.229 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:49.227+0000 7faa9107e0c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-05-12T09:32:54.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-05-12T09:32:54.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:32:54.106 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:32:54.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:32:54.107 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:32:54.622 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:32:54.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:32:54.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:32:54.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:32:54.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:32:54.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:32:54.643 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:54.640+0000 7f08bcaa50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:54.644 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:54.642+0000 7f08bcaa50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:54.645 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:54.643+0000 7f08bcaa50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:54.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:32:54.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:32:54.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:32:54.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:32:54.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:32:55.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:32:55.000 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:32:55.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:32:55.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:32:55.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:32:55.169 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:55.168+0000 7f08bcaa50c0 -1 Falling back to public interface 2021-05-12T09:32:55.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:32:56.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:32:56.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:32:56.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:32:56.405 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:32:56.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:32:56.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:32:56.790 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:32:57.391 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:57.388+0000 7f08bcaa50c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:32:57.790 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:32:57.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:32:57.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:32:57.792 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:32:57.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:32:57.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:32:58.180 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/177013,v1:127.0.0.1:6811/177013] [v2:127.0.0.1:6812/177013,v1:127.0.0.1:6813/177013] exists,up a2cad106-a33f-488b-9db6-e9a2fe7865cc 2021-05-12T09:32:58.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:32:58.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:32:58.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:32:58.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:65: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 2 2021-05-12T09:32:58.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T09:32:58.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:32:58.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T09:32:58.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:32:58.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-05-12T09:32:58.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T09:32:58.187 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:32:58.187 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:32:58.187 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:32:58.187 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-05-12T09:32:58.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-05-12T09:32:58.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:32:58.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:32:58.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T09:32:58.188 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:32:58.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:32:58.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:32:58.189 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:32:58.189 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:32:58.190 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:32:58.190 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:32:58.190 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:32:58.190 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:32:58.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:32:58.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:32:58.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T09:32:58.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T09:32:58.192 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:32:58.192 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:32:58.192 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:32:58.192 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:32:58.192 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:32:58.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/2 2021-05-12T09:32:58.193 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:32:58.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=f2e0dc56-605e-498b-8d04-0b112c7819b4 2021-05-12T09:32:58.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 f2e0dc56-605e-498b-8d04-0b112c7819b4' 2021-05-12T09:32:58.194 INFO:tasks.workunit.client.0.smithi028.stdout:add osd2 f2e0dc56-605e-498b-8d04-0b112c7819b4 2021-05-12T09:32:58.194 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:32:58.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDKoJtghqSRCxAA0ac2XMIFc3HoypSACgYGGg== 2021-05-12T09:32:58.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDKoJtghqSRCxAA0ac2XMIFc3HoypSACgYGGg=="}' 2021-05-12T09:32:58.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new f2e0dc56-605e-498b-8d04-0b112c7819b4 -i td/osd-markdown/2/new.json 2021-05-12T09:32:58.821 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:32:58.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/2/new.json 2021-05-12T09:32:58.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDKoJtghqSRCxAA0ac2XMIFc3HoypSACgYGGg== --osd-uuid f2e0dc56-605e-498b-8d04-0b112c7819b4 2021-05-12T09:32:58.855 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:58.854+0000 7f06495510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:58.857 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:58.856+0000 7f06495510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:58.859 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:58.857+0000 7f06495510c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:32:58.859 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:58.858+0000 7f06495510c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T09:32:58.860 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:58.858+0000 7f06495510c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T09:32:58.860 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:58.858+0000 7f06495510c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T09:32:58.904 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:32:58.900+0000 7f06495510c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-05-12T09:33:03.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-05-12T09:33:03.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:33:03.725 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd2 key to auth repository 2021-05-12T09:33:03.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T09:33:03.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:33:04.233 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:33:04.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T09:33:04.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:33:04.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:33:04.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:33:04.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:33:04.252 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:33:04.249+0000 7f1f3de280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:33:04.254 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:33:04.252+0000 7f1f3de280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:33:04.256 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:33:04.254+0000 7f1f3de280c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:33:04.620 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T09:33:04.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:33:04.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:33:04.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:33:04.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:33:04.622 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:33:04.622 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:33:04.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:33:04.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:33:04.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:33:04.776 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:33:04.774+0000 7f1f3de280c0 -1 Falling back to public interface 2021-05-12T09:33:04.996 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:33:05.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:33:05.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:33:05.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:33:05.998 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:33:05.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:33:05.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:33:06.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:33:07.019 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:33:07.017+0000 7f1f3de280c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T09:33:07.403 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:33:07.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:33:07.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:33:07.404 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:33:07.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:33:07.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:33:07.790 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/178406,v1:127.0.0.1:6819/178406] [v2:127.0.0.1:6820/178406,v1:127.0.0.1:6821/178406] exists,up f2e0dc56-605e-498b-8d04-0b112c7819b4 2021-05-12T09:33:07.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:33:07.791 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:33:07.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:33:07.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:67: TEST_markdown_exceed_maxdown_count: create_rbd_pool 2021-05-12T09:33:07.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-05-12T09:33:08.236 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'rbd' does not exist 2021-05-12T09:33:08.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-05-12T09:33:08.246 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-05-12T09:33:09.315 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'rbd' already exists 2021-05-12T09:33:09.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:33:10.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-05-12T09:33:12.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:70: TEST_markdown_exceed_maxdown_count: local count=3 2021-05-12T09:33:12.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:71: TEST_markdown_exceed_maxdown_count: local sleeptime=10 2021-05-12T09:33:12.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:72: TEST_markdown_exceed_maxdown_count: local period=300 2021-05-12T09:33:12.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:73: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-05-12T09:33:12.444 INFO:tasks.workunit.client.0.smithi028.stdout:{} 2021-05-12T09:33:12.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:74: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_period 300' 2021-05-12T09:33:12.578 INFO:tasks.workunit.client.0.smithi028.stdout:{} 2021-05-12T09:33:12.589 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:76: TEST_markdown_exceed_maxdown_count: markdown_N_impl 4 300 10 2021-05-12T09:33:12.589 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2021-05-12T09:33:12.589 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=300 2021-05-12T09:33:12.589 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-05-12T09:33:12.590 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2021-05-12T09:33:12.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T09:33:12.591 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T09:33:12.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T09:33:13.095 INFO:tasks.workunit.client.0.smithi028.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T09:33:13.096 INFO:tasks.workunit.client.0.smithi028.stdout:-1 0.29306 root default 2021-05-12T09:33:13.096 INFO:tasks.workunit.client.0.smithi028.stdout:-3 0.29306 host smithi028 2021-05-12T09:33:13.097 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:33:13.097 INFO:tasks.workunit.client.0.smithi028.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T09:33:13.098 INFO:tasks.workunit.client.0.smithi028.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T09:33:13.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T09:33:13.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T09:33:13.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T09:33:13.487 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:33:13.488 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T09:33:13.488 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T09:33:14.188 INFO:tasks.workunit.client.0.smithi028.stderr:marked down osd.0. 2021-05-12T09:33:14.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T09:33:24.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T09:33:24.201 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T09:33:24.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T09:33:24.705 INFO:tasks.workunit.client.0.smithi028.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T09:33:24.706 INFO:tasks.workunit.client.0.smithi028.stdout:-1 0.29306 root default 2021-05-12T09:33:24.706 INFO:tasks.workunit.client.0.smithi028.stdout:-3 0.29306 host smithi028 2021-05-12T09:33:24.706 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:33:24.706 INFO:tasks.workunit.client.0.smithi028.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T09:33:24.707 INFO:tasks.workunit.client.0.smithi028.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T09:33:24.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T09:33:24.717 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T09:33:24.718 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T09:33:25.109 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:33:25.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T09:33:25.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T09:33:25.502 INFO:tasks.workunit.client.0.smithi028.stderr:marked down osd.0. 2021-05-12T09:33:25.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T09:33:35.513 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T09:33:35.513 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T09:33:35.656 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T09:33:36.023 INFO:tasks.workunit.client.0.smithi028.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T09:33:36.025 INFO:tasks.workunit.client.0.smithi028.stdout:-1 0.29306 root default 2021-05-12T09:33:36.025 INFO:tasks.workunit.client.0.smithi028.stdout:-3 0.29306 host smithi028 2021-05-12T09:33:36.026 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:33:36.026 INFO:tasks.workunit.client.0.smithi028.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T09:33:36.027 INFO:tasks.workunit.client.0.smithi028.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T09:33:36.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T09:33:36.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T09:33:36.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T09:33:36.437 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:33:36.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T09:33:36.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T09:33:36.860 INFO:tasks.workunit.client.0.smithi028.stderr:marked down osd.0. 2021-05-12T09:33:36.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T09:33:46.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-05-12T09:33:46.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-05-12T09:33:47.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-05-12T09:33:47.378 INFO:tasks.workunit.client.0.smithi028.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-05-12T09:33:47.378 INFO:tasks.workunit.client.0.smithi028.stdout:-1 0.29306 root default 2021-05-12T09:33:47.378 INFO:tasks.workunit.client.0.smithi028.stdout:-3 0.29306 host smithi028 2021-05-12T09:33:47.379 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:33:47.379 INFO:tasks.workunit.client.0.smithi028.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-05-12T09:33:47.379 INFO:tasks.workunit.client.0.smithi028.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-05-12T09:33:47.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-05-12T09:33:47.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-05-12T09:33:47.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-05-12T09:33:47.885 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:33:47.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-05-12T09:33:47.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-05-12T09:33:48.299 INFO:tasks.workunit.client.0.smithi028.stderr:marked down osd.0. 2021-05-12T09:33:48.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-05-12T09:33:58.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: ceph osd tree 2021-05-12T09:33:58.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep down 2021-05-12T09:33:58.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep osd.0 2021-05-12T09:33:58.701 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2021-05-12T09:33:58.703 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-05-12T09:33:58.703 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T09:33:58.703 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:33:58.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T09:33:58.704 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:33:58.704 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:33:58.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:33:58.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:33:58.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:33:58.810 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:33:58.810 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:33:58.811 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:33:58.811 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:33:58.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:33:58.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:33:58.813 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:33:58.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:33:58.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:33:58.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:33:58.815 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:33:58.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:33:58.816 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:33:58.816 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T09:33:58.823 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:33:58.824 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:33:58.824 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:33:58.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164527 2021-05-12T09:33:58.826 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:33:58.826 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:33:58.826 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-05-12T09:33:58.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-05-12T09:33:58.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-05-12T09:33:58.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-05-12T09:33:58.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T09:33:58.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:33:58.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T09:33:58.828 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:33:58.828 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:33:58.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:33:58.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:33:58.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:33:58.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:33:58.831 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:33:58.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:33:58.831 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:33:58.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:33:58.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:33:58.832 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:33:58.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:33:58.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:33:58.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:33:58.833 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:33:58.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:33:58.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:33:58.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T09:33:58.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:33:58.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:33:58.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:33:58.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164527 2021-05-12T09:33:58.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:33:58.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:33:58.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-05-12T09:33:58.838 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:33:58.839 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:33:58.839 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:33:58.839 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.164527 2021-05-12T09:33:58.839 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:33:58.840 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-05-12T09:33:58.840 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:33:58.840 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-05-12T09:33:58.840 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_osd_stop td/osd-markdown 2021-05-12T09:33:58.840 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:132: TEST_osd_stop: local dir=td/osd-markdown 2021-05-12T09:33:58.841 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:134: TEST_osd_stop: run_mon td/osd-markdown a 2021-05-12T09:33:58.841 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-05-12T09:33:58.841 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:33:58.841 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:33:58.842 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:33:58.842 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-05-12T09:33:58.842 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-05-12T09:33:59.196 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:33:59.197 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:33:59.197 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:33:59.197 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:33:59.197 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:33:59.198 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:33:59.198 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:33:59.198 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:33:59.447 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:33:59.449 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:33:59.449 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:33:59.449 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:33:59.450 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:33:59.450 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:33:59.450 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:33:59.450 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:33:59.450 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:33:59.451 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:33:59.451 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:33:59.451 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:33:59.451 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164527/ceph-mon.a.asok 2021-05-12T09:33:59.452 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:33:59.452 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164527/ceph-mon.a.asok config get fsid 2021-05-12T09:33:59.558 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:33:59.558 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:33:59.558 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:33:59.559 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:33:59.559 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:33:59.559 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:33:59.559 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:33:59.560 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:33:59.560 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:33:59.560 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:33:59.560 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:33:59.561 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164527/ceph-mon.a.asok 2021-05-12T09:33:59.561 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:33:59.561 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.164527/ceph-mon.a.asok config get mon_host 2021-05-12T09:33:59.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:135: TEST_osd_stop: run_mgr td/osd-markdown x 2021-05-12T09:33:59.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-05-12T09:33:59.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:33:59.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:33:59.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:33:59.659 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-05-12T09:33:59.659 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:34:00.545 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:34:00.545 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:34:00.545 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:34:00.545 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:34:00.546 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:00.546 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:34:00.546 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:34:00.547 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:34:00.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:34:00.572 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:136: TEST_osd_stop: run_osd td/osd-markdown 0 2021-05-12T09:34:00.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T09:34:00.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:34:00.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:34:00.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:34:00.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-05-12T09:34:00.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T09:34:00.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:34:00.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:34:00.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:34:00.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-05-12T09:34:00.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-05-12T09:34:00.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:34:00.576 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:34:00.576 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T09:34:00.576 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:34:00.576 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:34:00.576 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:34:00.577 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:34:00.577 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:00.577 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:34:00.577 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:34:00.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:34:00.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:34:00.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:34:00.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:34:00.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T09:34:00.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T09:34:00.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:34:00.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:34:00.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:34:00.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:34:00.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:34:00.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/0 2021-05-12T09:34:00.580 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:34:00.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=473fd4a5-537f-48dd-95b4-1727913e7cd3 2021-05-12T09:34:00.581 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 473fd4a5-537f-48dd-95b4-1727913e7cd3 2021-05-12T09:34:00.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 473fd4a5-537f-48dd-95b4-1727913e7cd3' 2021-05-12T09:34:00.582 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:34:00.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAIoZtgYiXOIhAApKgXAuYTiL3gxfNfC/q82Q== 2021-05-12T09:34:00.586 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAIoZtgYiXOIhAApKgXAuYTiL3gxfNfC/q82Q=="}' 2021-05-12T09:34:00.586 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 473fd4a5-537f-48dd-95b4-1727913e7cd3 -i td/osd-markdown/0/new.json 2021-05-12T09:34:00.913 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:34:00.924 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/0/new.json 2021-05-12T09:34:00.925 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAIoZtgYiXOIhAApKgXAuYTiL3gxfNfC/q82Q== --osd-uuid 473fd4a5-537f-48dd-95b4-1727913e7cd3 2021-05-12T09:34:00.947 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:00.945+0000 7f1aaef360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:00.949 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:00.947+0000 7f1aaef360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:00.950 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:00.948+0000 7f1aaef360c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:00.951 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:00.950+0000 7f1aaef360c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T09:34:00.952 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:00.950+0000 7f1aaef360c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T09:34:00.952 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:00.950+0000 7f1aaef360c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-05-12T09:34:00.995 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:00.993+0000 7f1aaef360c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-05-12T09:34:05.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-05-12T09:34:05.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:34:05.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:34:05.702 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:34:05.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:34:06.197 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:34:06.198 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:34:06.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:34:06.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:34:06.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:34:06.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:34:06.217 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:06.215+0000 7f8ad3d4d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:06.219 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:06.217+0000 7f8ad3d4d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:06.220 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:06.219+0000 7f8ad3d4d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:06.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:34:06.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:34:06.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:34:06.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:34:06.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:34:06.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:34:06.576 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:34:06.576 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:34:06.577 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:34:06.577 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:34:06.736 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:06.734+0000 7f8ad3d4d0c0 -1 Falling back to public interface 2021-05-12T09:34:06.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:34:07.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:34:07.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:34:07.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:34:07.962 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:34:07.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:34:07.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:34:08.346 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:34:08.947 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:08.945+0000 7f8ad3d4d0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:34:09.347 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:34:09.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:34:09.349 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:34:09.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:34:09.350 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:34:09.350 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:34:09.734 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/180642,v1:127.0.0.1:6803/180642] [v2:127.0.0.1:6804/180642,v1:127.0.0.1:6805/180642] exists,up 473fd4a5-537f-48dd-95b4-1727913e7cd3 2021-05-12T09:34:09.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:34:09.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:34:09.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:34:09.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:137: TEST_osd_stop: run_osd td/osd-markdown 1 2021-05-12T09:34:09.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T09:34:09.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:34:09.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:34:09.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:34:09.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-05-12T09:34:09.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T09:34:09.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:34:09.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:34:09.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:34:09.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-05-12T09:34:09.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-05-12T09:34:09.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:34:09.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:34:09.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T09:34:09.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:34:09.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:34:09.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:34:09.744 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:34:09.745 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:09.745 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:34:09.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:34:09.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:34:09.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:34:09.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:34:09.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:34:09.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T09:34:09.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T09:34:09.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:34:09.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:34:09.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:34:09.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:34:09.748 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:34:09.748 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/1 2021-05-12T09:34:09.749 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:34:09.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=11028249-e538-4559-b00d-3191ec803f61 2021-05-12T09:34:09.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 11028249-e538-4559-b00d-3191ec803f61' 2021-05-12T09:34:09.750 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 11028249-e538-4559-b00d-3191ec803f61 2021-05-12T09:34:09.751 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:34:09.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQARoZtgJtGkLBAAg63bZiRNsY4cc9c7nNOaiQ== 2021-05-12T09:34:09.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQARoZtgJtGkLBAAg63bZiRNsY4cc9c7nNOaiQ=="}' 2021-05-12T09:34:09.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 11028249-e538-4559-b00d-3191ec803f61 -i td/osd-markdown/1/new.json 2021-05-12T09:34:10.323 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:34:10.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/1/new.json 2021-05-12T09:34:10.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQARoZtgJtGkLBAAg63bZiRNsY4cc9c7nNOaiQ== --osd-uuid 11028249-e538-4559-b00d-3191ec803f61 2021-05-12T09:34:10.358 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:10.355+0000 7ffa0331d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:10.359 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:10.356+0000 7ffa0331d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:10.360 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:10.358+0000 7ffa0331d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:10.361 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:10.359+0000 7ffa0331d0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T09:34:10.361 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:10.359+0000 7ffa0331d0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T09:34:10.362 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:10.359+0000 7ffa0331d0c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-05-12T09:34:10.432 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:10.429+0000 7ffa0331d0c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-05-12T09:34:15.253 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-05-12T09:34:15.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:34:15.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:34:15.255 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:34:15.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:34:15.769 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:34:15.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:34:15.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:34:15.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:34:15.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:34:15.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:34:15.789 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:15.786+0000 7fe2f44950c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:15.791 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:15.788+0000 7fe2f44950c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:15.792 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:15.789+0000 7fe2f44950c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:16.146 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:34:16.147 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:34:16.147 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:34:16.147 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:34:16.147 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:34:16.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:34:16.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:34:16.148 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:34:16.149 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:34:16.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:34:16.309 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:16.306+0000 7fe2f44950c0 -1 Falling back to public interface 2021-05-12T09:34:16.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:34:17.528 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:34:17.924 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:34:17.925 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:34:17.926 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:34:17.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:34:17.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:34:17.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:34:18.484 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:18.481+0000 7fe2f44950c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:34:18.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:34:18.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:34:18.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:34:18.922 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:34:18.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:34:18.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:34:19.336 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/182027,v1:127.0.0.1:6811/182027] [v2:127.0.0.1:6812/182027,v1:127.0.0.1:6813/182027] exists,up 11028249-e538-4559-b00d-3191ec803f61 2021-05-12T09:34:19.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:34:19.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:34:19.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:34:19.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:138: TEST_osd_stop: run_osd td/osd-markdown 2 2021-05-12T09:34:19.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-05-12T09:34:19.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:34:19.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-05-12T09:34:19.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:34:19.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-05-12T09:34:19.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-05-12T09:34:19.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:34:19.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:34:19.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:34:19.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-05-12T09:34:19.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-05-12T09:34:19.346 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:34:19.346 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:34:19.346 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-05-12T09:34:19.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:34:19.347 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:34:19.347 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:34:19.347 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:34:19.347 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:19.347 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:34:19.348 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:34:19.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' 2021-05-12T09:34:19.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:34:19.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:34:19.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:34:19.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-05-12T09:34:19.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-05-12T09:34:19.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:34:19.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:34:19.350 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:34:19.350 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:34:19.351 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:34:19.351 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-markdown/2 2021-05-12T09:34:19.351 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:34:19.351 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=3d6d8906-7689-43cd-b6ca-cd2ee2e0fc47 2021-05-12T09:34:19.352 INFO:tasks.workunit.client.0.smithi028.stdout:add osd2 3d6d8906-7689-43cd-b6ca-cd2ee2e0fc47 2021-05-12T09:34:19.353 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 3d6d8906-7689-43cd-b6ca-cd2ee2e0fc47' 2021-05-12T09:34:19.353 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:34:19.353 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAboZtgK2nrFBAAJ2KId+Zip9hHvsPKvGRzvQ== 2021-05-12T09:34:19.353 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAboZtgK2nrFBAAJ2KId+Zip9hHvsPKvGRzvQ=="}' 2021-05-12T09:34:19.355 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 3d6d8906-7689-43cd-b6ca-cd2ee2e0fc47 -i td/osd-markdown/2/new.json 2021-05-12T09:34:19.841 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:34:19.853 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-markdown/2/new.json 2021-05-12T09:34:19.854 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAboZtgK2nrFBAAJ2KId+Zip9hHvsPKvGRzvQ== --osd-uuid 3d6d8906-7689-43cd-b6ca-cd2ee2e0fc47 2021-05-12T09:34:19.875 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:19.872+0000 7f61bb1cc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:19.877 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:19.874+0000 7f61bb1cc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:19.878 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:19.876+0000 7f61bb1cc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:19.879 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:19.876+0000 7f61bb1cc0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T09:34:19.879 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:19.876+0000 7f61bb1cc0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T09:34:19.880 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:19.876+0000 7f61bb1cc0c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-05-12T09:34:19.920 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:19.918+0000 7f61bb1cc0c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-05-12T09:34:24.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-05-12T09:34:24.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:34:24.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-05-12T09:34:24.542 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd2 key to auth repository 2021-05-12T09:34:24.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:34:25.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-05-12T09:34:25.052 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.2 2021-05-12T09:34:25.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=77810cca-72e9-493c-adaa-89e482482e29 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.164527/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:34:25.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:34:25.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:34:25.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:34:25.071 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:25.068+0000 7effee8ec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:25.073 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:25.070+0000 7effee8ec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:25.075 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:25.072+0000 7effee8ec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:25.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-05-12T09:34:25.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:34:25.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=2 2021-05-12T09:34:25.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:34:25.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:34:25.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:34:25.433 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:34:25.433 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:34:25.434 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:34:25.434 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:34:25.602 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:25.600+0000 7effee8ec0c0 -1 Falling back to public interface 2021-05-12T09:34:25.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:34:26.857 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:34:26.858 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:34:26.858 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:34:26.858 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:34:26.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:34:26.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:34:27.247 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:34:27.914 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:27.912+0000 7effee8ec0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-05-12T09:34:28.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:34:28.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:34:28.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:34:28.250 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:34:28.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:34:28.251 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.2 up' 2021-05-12T09:34:28.636 INFO:tasks.workunit.client.0.smithi028.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/183412,v1:127.0.0.1:6819/183412] [v2:127.0.0.1:6820/183412,v1:127.0.0.1:6821/183412] exists,up 3d6d8906-7689-43cd-b6ca-cd2ee2e0fc47 2021-05-12T09:34:28.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:34:28.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:34:28.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:34:28.638 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: cat td/osd-markdown/osd.0.pid 2021-05-12T09:34:28.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: osd_0_pid=180642 2021-05-12T09:34:28.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:140: TEST_osd_stop: ps -p 180642 2021-05-12T09:34:28.644 INFO:tasks.workunit.client.0.smithi028.stdout: PID TTY TIME CMD 2021-05-12T09:34:28.644 INFO:tasks.workunit.client.0.smithi028.stdout: 180642 ? 00:00:00 ceph-osd 2021-05-12T09:34:28.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: ceph osd tree 2021-05-12T09:34:28.645 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep osd.0 2021-05-12T09:34:28.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep up 2021-05-12T09:34:29.036 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-05-12T09:34:29.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop: ceph osd stop osd.0 2021-05-12T09:34:29.680 INFO:tasks.workunit.client.0.smithi028.stderr:osd.0 is already stopped. 2021-05-12T09:34:29.689 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop: sleep 15 2021-05-12T09:34:44.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: ceph osd tree 2021-05-12T09:34:44.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep down 2021-05-12T09:34:44.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep osd.0 2021-05-12T09:34:45.080 INFO:tasks.workunit.client.0.smithi028.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2021-05-12T09:34:45.081 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop: ps -p 180642 2021-05-12T09:34:45.086 INFO:tasks.workunit.client.0.smithi028.stdout: PID TTY TIME CMD 2021-05-12T09:34:45.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-05-12T09:34:45.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T09:34:45.088 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:34:45.088 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T09:34:45.088 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:34:45.088 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:34:45.088 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:34:45.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:34:45.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:34:45.194 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:34:45.194 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:34:45.195 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:34:45.195 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:34:45.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:34:45.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:34:45.196 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:34:45.198 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:34:45.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:34:45.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:34:45.199 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:34:45.200 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:34:45.200 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:34:45.200 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T09:34:45.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:34:45.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:45.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:34:45.206 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164527 2021-05-12T09:34:45.207 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:34:45.208 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:34:45.208 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T09:34:45.208 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-markdown 0 2021-05-12T09:34:45.208 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-05-12T09:34:45.209 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T09:34:45.209 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-05-12T09:34:45.209 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:34:45.209 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:34:45.210 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:34:45.210 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:34:45.210 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:34:45.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:34:45.212 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:34:45.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:34:45.213 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:34:45.213 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:34:45.213 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:34:45.213 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:34:45.214 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:34:45.214 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:34:45.215 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:34:45.216 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:34:45.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:34:45.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T09:34:45.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-05-12T09:34:45.217 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:34:45.217 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:45.217 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164527 2021-05-12T09:34:45.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164527 2021-05-12T09:34:45.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:34:45.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:34:45.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T09:34:45.220 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T09:34:45.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T09:34:45.255 INFO:tasks.workunit:Running workunit osd/osd-reactivate.sh... 2021-05-12T09:34:45.255 DEBUG:teuthology.orchestra.run.smithi028:workunit test osd/osd-reactivate.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2021-05-12T09:34:45.315 INFO:tasks.workunit.client.0.smithi028.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T09:34:45.323 INFO:tasks.workunit.client.0.smithi028.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T09:34:45.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:34:45.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:34:45.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T09:34:45.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T09:34:45.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T09:34:45.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T09:34:45.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T09:34:45.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T09:34:45.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-reactivate 2021-05-12T09:34:45.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:19: run: local dir=td/osd-reactivate 2021-05-12T09:34:45.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:20: run: shift 2021-05-12T09:34:45.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: export CEPH_MON=127.0.0.1:7122 2021-05-12T09:34:45.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: CEPH_MON=127.0.0.1:7122 2021-05-12T09:34:45.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:23: run: export CEPH_ARGS 2021-05-12T09:34:45.328 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: uuidgen 2021-05-12T09:34:45.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: CEPH_ARGS+='--fsid=cab08bfb-00c4-48ad-8725-8f7bef4361b9 --auth-supported=none ' 2021-05-12T09:34:45.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:25: run: CEPH_ARGS+='--mon-host=127.0.0.1:7122 ' 2021-05-12T09:34:45.329 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: set 2021-05-12T09:34:45.329 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T09:34:45.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: local funcs=TEST_reactivate 2021-05-12T09:34:45.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:28: run: for func in $funcs 2021-05-12T09:34:45.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:29: run: setup td/osd-reactivate 2021-05-12T09:34:45.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reactivate 2021-05-12T09:34:45.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reactivate 2021-05-12T09:34:45.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-05-12T09:34:45.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:34:45.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-05-12T09:34:45.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:34:45.338 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:34:45.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:34:45.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:34:45.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:34:45.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:34:45.339 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:34:45.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:34:45.340 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:34:45.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:34:45.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:34:45.340 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:34:45.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:34:45.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:34:45.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:34:45.341 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:34:45.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:34:45.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:34:45.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-05-12T09:34:45.342 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:34:45.343 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:45.343 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184120 2021-05-12T09:34:45.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.184120 2021-05-12T09:34:45.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:34:45.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:34:45.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reactivate 2021-05-12T09:34:45.344 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:34:45.344 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:45.344 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184120 2021-05-12T09:34:45.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.184120 2021-05-12T09:34:45.347 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:34:45.347 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T09:34:45.347 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T09:34:45.347 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:34:45.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reactivate 1' TERM HUP INT 2021-05-12T09:34:45.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:30: run: TEST_reactivate td/osd-reactivate 2021-05-12T09:34:45.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:36: TEST_reactivate: local dir=td/osd-reactivate 2021-05-12T09:34:45.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:38: TEST_reactivate: run_mon td/osd-reactivate a 2021-05-12T09:34:45.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reactivate 2021-05-12T09:34:45.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:34:45.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:34:45.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:34:45.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reactivate/a 2021-05-12T09:34:45.350 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reactivate/a --run-dir=td/osd-reactivate 2021-05-12T09:34:45.673 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:34:45.674 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:34:45.674 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:34:45.674 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:34:45.674 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:45.675 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184120 2021-05-12T09:34:45.675 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184120/$cluster-$name.asok' 2021-05-12T09:34:45.675 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reactivate/a '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.184120/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reactivate/log --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:34:45.933 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:34:45.935 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:34:45.936 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:34:45.936 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:34:45.936 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:34:45.936 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:34:45.937 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:34:45.937 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:34:45.937 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:34:45.937 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:34:45.937 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:45.938 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184120 2021-05-12T09:34:45.938 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.184120/ceph-mon.a.asok 2021-05-12T09:34:45.938 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:34:45.938 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.184120/ceph-mon.a.asok config get fsid 2021-05-12T09:34:46.037 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:34:46.037 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:34:46.038 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:34:46.038 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:34:46.038 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:34:46.038 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:34:46.039 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:34:46.039 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:34:46.039 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:34:46.039 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:46.040 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184120 2021-05-12T09:34:46.040 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.184120/ceph-mon.a.asok 2021-05-12T09:34:46.040 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:34:46.040 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.184120/ceph-mon.a.asok config get mon_host 2021-05-12T09:34:46.135 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:39: TEST_reactivate: run_mgr td/osd-reactivate x 2021-05-12T09:34:46.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reactivate 2021-05-12T09:34:46.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:34:46.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:34:46.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:34:46.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reactivate/x 2021-05-12T09:34:46.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:34:47.030 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:34:47.031 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:34:47.031 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:34:47.031 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:34:47.032 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:47.032 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184120 2021-05-12T09:34:47.032 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184120/$cluster-$name.asok' 2021-05-12T09:34:47.032 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:34:47.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reactivate/x '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.184120/$cluster-$name.asok' --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:34:47.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:40: TEST_reactivate: run_osd td/osd-reactivate 0 2021-05-12T09:34:47.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reactivate 2021-05-12T09:34:47.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:34:47.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:34:47.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:34:47.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reactivate/0 2021-05-12T09:34:47.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cab08bfb-00c4-48ad-8725-8f7bef4361b9 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2021-05-12T09:34:47.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:34:47.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:34:47.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:34:47.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2021-05-12T09:34:47.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2021-05-12T09:34:47.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:34:47.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:34:47.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2021-05-12T09:34:47.060 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:34:47.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:34:47.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:34:47.061 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:34:47.061 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:47.061 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184120 2021-05-12T09:34:47.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184120/$cluster-$name.asok' 2021-05-12T09:34:47.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.184120/$cluster-$name.asok' 2021-05-12T09:34:47.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:34:47.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:34:47.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:34:47.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2021-05-12T09:34:47.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2021-05-12T09:34:47.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:34:47.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:34:47.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:34:47.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:34:47.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:34:47.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-reactivate/0 2021-05-12T09:34:47.065 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:34:47.065 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=7eba2502-2a67-44ff-90ef-27fa712601d2 2021-05-12T09:34:47.065 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 7eba2502-2a67-44ff-90ef-27fa712601d2' 2021-05-12T09:34:47.066 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 7eba2502-2a67-44ff-90ef-27fa712601d2 2021-05-12T09:34:47.066 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:34:47.069 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQA3oZtg6gIFBBAAymWMlYfjUFYqrkvaXCT7TQ== 2021-05-12T09:34:47.069 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQA3oZtg6gIFBBAAymWMlYfjUFYqrkvaXCT7TQ=="}' 2021-05-12T09:34:47.070 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 7eba2502-2a67-44ff-90ef-27fa712601d2 -i td/osd-reactivate/0/new.json 2021-05-12T09:34:47.392 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:34:47.403 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-reactivate/0/new.json 2021-05-12T09:34:47.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=cab08bfb-00c4-48ad-8725-8f7bef4361b9 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.184120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA3oZtg6gIFBBAAymWMlYfjUFYqrkvaXCT7TQ== --osd-uuid 7eba2502-2a67-44ff-90ef-27fa712601d2 2021-05-12T09:34:47.426 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:47.423+0000 7fdedb6f80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:47.427 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:47.425+0000 7fdedb6f80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:47.428 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:47.426+0000 7fdedb6f80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:47.429 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:47.426+0000 7fdedb6f80c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-05-12T09:34:47.429 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:47.426+0000 7fdedb6f80c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-05-12T09:34:47.429 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:47.426+0000 7fdedb6f80c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-05-12T09:34:47.464 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:47.462+0000 7fdedb6f80c0 -1 bluestore(td/osd-reactivate/0) _read_fsid unparsable uuid 2021-05-12T09:34:52.107 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-reactivate/0/keyring 2021-05-12T09:34:52.107 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:34:52.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:34:52.108 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:34:52.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-reactivate/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:34:52.610 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:34:52.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:34:52.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=cab08bfb-00c4-48ad-8725-8f7bef4361b9 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.184120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:34:52.612 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:34:52.612 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:34:52.612 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:34:52.630 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:52.627+0000 7f71aa3bb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:52.632 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:52.629+0000 7f71aa3bb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:52.633 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:52.630+0000 7f71aa3bb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:52.988 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:34:52.988 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:34:52.988 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:34:52.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:34:52.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:34:52.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:34:52.990 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:34:52.990 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:34:52.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:34:52.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:34:53.155 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:53.153+0000 7f71aa3bb0c0 -1 Falling back to public interface 2021-05-12T09:34:53.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:34:54.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:34:54.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:34:54.386 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:34:54.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:34:54.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:34:54.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:34:54.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:34:55.381 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:55.379+0000 7f71aa3bb0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:34:55.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:34:55.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:34:55.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:34:55.776 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:34:55.776 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:34:55.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:34:56.161 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/185240,v1:127.0.0.1:6803/185240] [v2:127.0.0.1:6804/185240,v1:127.0.0.1:6805/185240] exists,up 7eba2502-2a67-44ff-90ef-27fa712601d2 2021-05-12T09:34:56.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:34:56.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:34:56.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:34:56.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:42: TEST_reactivate: kill_daemons td/osd-reactivate TERM osd 2021-05-12T09:34:56.163 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:34:56.163 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:34:56.164 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:34:56.164 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:34:56.164 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:34:56.267 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:34:56.267 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:44: TEST_reactivate: ready_path=td/osd-reactivate/0/ready 2021-05-12T09:34:56.267 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:45: TEST_reactivate: activate_path=td/osd-reactivate/0/active 2021-05-12T09:34:56.267 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:47: TEST_reactivate: rm -rf td/osd-reactivate/0/ready td/osd-reactivate/0/active 2021-05-12T09:34:56.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:48: TEST_reactivate: activate_osd td/osd-reactivate 0 2021-05-12T09:34:56.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local dir=td/osd-reactivate 2021-05-12T09:34:56.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2021-05-12T09:34:56.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local id=0 2021-05-12T09:34:56.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-05-12T09:34:56.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local osd_data=td/osd-reactivate/0 2021-05-12T09:34:56.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local 'ceph_args=--fsid=cab08bfb-00c4-48ad-8725-8f7bef4361b9 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2021-05-12T09:34:56.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:34:56.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:34:56.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:34:56.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2021-05-12T09:34:56.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2021-05-12T09:34:56.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --chdir=' 2021-05-12T09:34:56.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+= 2021-05-12T09:34:56.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2021-05-12T09:34:56.275 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: get_asok_path 2021-05-12T09:34:56.275 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:34:56.275 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:34:56.275 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:34:56.275 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:56.276 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184120 2021-05-12T09:34:56.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.184120/$cluster-$name.asok' 2021-05-12T09:34:56.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.184120/$cluster-$name.asok' 2021-05-12T09:34:56.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:34:56.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2021-05-12T09:34:56.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2021-05-12T09:34:56.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:34:56.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:34:56.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:34:56.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' ' 2021-05-12T09:34:56.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+= 2021-05-12T09:34:56.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: mkdir -p td/osd-reactivate/0 2021-05-12T09:34:56.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: echo start osd.0 2021-05-12T09:34:56.279 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:34:56.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: ceph-osd -i 0 --fsid=cab08bfb-00c4-48ad-8725-8f7bef4361b9 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.184120/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:34:56.280 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: cat td/osd-reactivate/0/whoami 2021-05-12T09:34:56.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: '[' 0 = 0 ']' 2021-05-12T09:34:56.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: ceph osd dump --format=json 2021-05-12T09:34:56.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: jq '.flags_set[]' 2021-05-12T09:34:56.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: grep -q '"noup"' 2021-05-12T09:34:56.289 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:56.287+0000 7fed899770c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:56.292 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:56.290+0000 7fed899770c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:56.294 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:56.291+0000 7fed899770c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:56.649 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: wait_for_osd up 0 2021-05-12T09:34:56.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:34:56.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:34:56.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:34:56.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:34:56.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:34:56.651 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:34:56.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:34:56.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:34:56.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:34:56.816 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:56.814+0000 7fed899770c0 -1 Falling back to public interface 2021-05-12T09:34:57.049 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/185240,v1:127.0.0.1:6803/185240] [v2:127.0.0.1:6804/185240,v1:127.0.0.1:6805/185240] exists,up 7eba2502-2a67-44ff-90ef-27fa712601d2 2021-05-12T09:34:57.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:34:57.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:34:57.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:34:57.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:31: run: teardown td/osd-reactivate 2021-05-12T09:34:57.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-05-12T09:34:57.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:34:57.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-05-12T09:34:57.053 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:34:57.053 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:34:57.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:34:57.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:34:57.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:34:57.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:34:57.156 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:34:57.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:34:57.157 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:34:57.158 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:34:57.158 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:34:57.159 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:34:57.160 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:34:57.160 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:34:57.160 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:34:57.161 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:34:57.161 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:34:57.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:34:57.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-05-12T09:34:57.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:34:57.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:57.165 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184120 2021-05-12T09:34:57.165 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.184120 2021-05-12T09:34:57.167 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:34:57.168 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:34:57.168 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: main: code=0 2021-05-12T09:34:57.168 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: teardown td/osd-reactivate 0 2021-05-12T09:34:57.169 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-05-12T09:34:57.169 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-05-12T09:34:57.169 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-05-12T09:34:57.169 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:34:57.170 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:34:57.170 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:34:57.170 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:34:57.171 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:34:57.171 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:34:57.171 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:34:57.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:34:57.173 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:34:57.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:34:57.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:34:57.174 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:34:57.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:34:57.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:34:57.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:34:57.175 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:34:57.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:34:57.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-05-12T09:34:57.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-05-12T09:34:57.177 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:34:57.178 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:57.178 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.184120 2021-05-12T09:34:57.178 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.184120 2021-05-12T09:34:57.178 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:34:57.179 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:34:57.179 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: main: return 0 2021-05-12T09:34:57.179 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-05-12T09:34:57.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-05-12T09:34:57.242 INFO:tasks.workunit:Running workunit osd/osd-recovery-prio.sh... 2021-05-12T09:34:57.242 DEBUG:teuthology.orchestra.run.smithi028:workunit test osd/osd-recovery-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cd0ad0001da189274219f5e829752717eb856d2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2021-05-12T09:34:57.303 INFO:tasks.workunit.client.0.smithi028.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-05-12T09:34:57.313 INFO:tasks.workunit.client.0.smithi028.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-05-12T09:34:57.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:34:57.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-05-12T09:34:57.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export PYTHONWARNINGS=ignore 2021-05-12T09:34:57.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: PYTHONWARNINGS=ignore 2021-05-12T09:34:57.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: export CEPH_CONF=/dev/null 2021-05-12T09:34:57.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: CEPH_CONF=/dev/null 2021-05-12T09:34:57.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: unset CEPH_ARGS 2021-05-12T09:34:57.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: local code 2021-05-12T09:34:57.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: run td/osd-recovery-prio 2021-05-12T09:34:57.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:21: run: local dir=td/osd-recovery-prio 2021-05-12T09:34:57.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:22: run: shift 2021-05-12T09:34:57.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2021-05-12T09:34:57.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2021-05-12T09:34:57.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:26: run: export CEPH_ARGS 2021-05-12T09:34:57.317 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: uuidgen 2021-05-12T09:34:57.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: CEPH_ARGS+='--fsid=87db7d12-ce3c-4a9b-8bb7-31c860ffeeb7 --auth-supported=none ' 2021-05-12T09:34:57.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T09:34:57.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:29: run: export objects=200 2021-05-12T09:34:57.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:29: run: objects=200 2021-05-12T09:34:57.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:30: run: export poolprefix=test 2021-05-12T09:34:57.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:30: run: poolprefix=test 2021-05-12T09:34:57.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: export FORCE_PRIO=255 2021-05-12T09:34:57.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: FORCE_PRIO=255 2021-05-12T09:34:57.319 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: export NORMAL_PRIO=190 2021-05-12T09:34:57.319 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: NORMAL_PRIO=190 2021-05-12T09:34:57.319 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: set 2021-05-12T09:34:57.319 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-05-12T09:34:57.322 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: local 'funcs=TEST_recovery_pool_priority 2021-05-12T09:34:57.322 INFO:tasks.workunit.client.0.smithi028.stderr:TEST_recovery_priority' 2021-05-12T09:34:57.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: for func in $funcs 2021-05-12T09:34:57.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:36: run: setup td/osd-recovery-prio 2021-05-12T09:34:57.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2021-05-12T09:34:57.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2021-05-12T09:34:57.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-05-12T09:34:57.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-05-12T09:34:57.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-05-12T09:34:57.324 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-05-12T09:34:57.324 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-05-12T09:34:57.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-05-12T09:34:57.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-05-12T09:34:57.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-05-12T09:34:57.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-05-12T09:34:57.325 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-05-12T09:34:57.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-05-12T09:34:57.326 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-05-12T09:34:57.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-05-12T09:34:57.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-05-12T09:34:57.326 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-05-12T09:34:57.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:34:57.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-05-12T09:34:57.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-05-12T09:34:57.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-05-12T09:34:57.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-05-12T09:34:57.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-05-12T09:34:57.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-05-12T09:34:57.331 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-05-12T09:34:57.331 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:57.331 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185947 2021-05-12T09:34:57.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.185947 2021-05-12T09:34:57.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-05-12T09:34:57.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-05-12T09:34:57.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2021-05-12T09:34:57.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-05-12T09:34:57.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:57.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185947 2021-05-12T09:34:57.333 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.185947 2021-05-12T09:34:57.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-05-12T09:34:57.334 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-05-12T09:34:57.334 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-05-12T09:34:57.334 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-05-12T09:34:57.334 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2021-05-12T09:34:57.334 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: TEST_recovery_pool_priority td/osd-recovery-prio 2021-05-12T09:34:57.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:323: TEST_recovery_pool_priority: local dir=td/osd-recovery-prio 2021-05-12T09:34:57.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:324: TEST_recovery_pool_priority: local pools=3 2021-05-12T09:34:57.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:325: TEST_recovery_pool_priority: local OSDS=2 2021-05-12T09:34:57.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:327: TEST_recovery_pool_priority: run_mon td/osd-recovery-prio a 2021-05-12T09:34:57.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2021-05-12T09:34:57.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-05-12T09:34:57.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-05-12T09:34:57.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-05-12T09:34:57.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2021-05-12T09:34:57.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2021-05-12T09:34:57.692 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-05-12T09:34:57.692 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:34:57.692 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:34:57.693 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:34:57.693 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:57.694 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185947 2021-05-12T09:34:57.694 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185947/$cluster-$name.asok' 2021-05-12T09:34:57.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.185947/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-05-12T09:34:57.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-05-12T09:34:57.954 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-05-12T09:34:57.954 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:34:57.954 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:34:57.955 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=fsid 2021-05-12T09:34:57.955 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .fsid 2021-05-12T09:34:57.955 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:34:57.955 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:34:57.955 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:34:57.956 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:34:57.956 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:57.956 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185947 2021-05-12T09:34:57.956 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185947/ceph-mon.a.asok 2021-05-12T09:34:57.957 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:34:57.957 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.185947/ceph-mon.a.asok config get fsid 2021-05-12T09:34:58.056 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-05-12T09:34:58.057 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local daemon=mon 2021-05-12T09:34:58.057 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local id=a 2021-05-12T09:34:58.057 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: local config=mon_host 2021-05-12T09:34:58.057 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: jq -r .mon_host 2021-05-12T09:34:58.058 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: get_asok_path mon.a 2021-05-12T09:34:58.058 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-05-12T09:34:58.058 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-05-12T09:34:58.058 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:34:58.059 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:58.059 INFO:tasks.workunit.client.0.smithi028.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185947 2021-05-12T09:34:58.059 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185947/ceph-mon.a.asok 2021-05-12T09:34:58.060 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: CEPH_ARGS= 2021-05-12T09:34:58.060 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: ceph --format json daemon /tmp/ceph-asok.185947/ceph-mon.a.asok config get mon_host 2021-05-12T09:34:58.155 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:328: TEST_recovery_pool_priority: run_mgr td/osd-recovery-prio x 2021-05-12T09:34:58.155 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2021-05-12T09:34:58.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-05-12T09:34:58.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-05-12T09:34:58.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-05-12T09:34:58.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2021-05-12T09:34:58.156 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-05-12T09:34:58.799 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-05-12T09:34:58.800 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:34:58.800 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:34:58.800 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:34:58.800 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:58.801 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185947 2021-05-12T09:34:58.801 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185947/$cluster-$name.asok' 2021-05-12T09:34:58.801 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:34:58.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.185947/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-05-12T09:34:58.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:329: TEST_recovery_pool_priority: export CEPH_ARGS 2021-05-12T09:34:58.824 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: expr 2 - 1 2021-05-12T09:34:58.825 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: seq 0 1 2021-05-12T09:34:58.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T09:34:58.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:333: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 0 2021-05-12T09:34:58.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T09:34:58.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:34:58.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-05-12T09:34:58.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:34:58.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2021-05-12T09:34:58.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=87db7d12-ce3c-4a9b-8bb7-31c860ffeeb7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T09:34:58.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:34:58.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:34:58.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:34:58.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2021-05-12T09:34:58.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2021-05-12T09:34:58.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:34:58.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:34:58.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T09:34:58.833 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:34:58.834 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:34:58.834 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:34:58.834 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:34:58.834 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:34:58.834 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185947 2021-05-12T09:34:58.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185947/$cluster-$name.asok' 2021-05-12T09:34:58.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.185947/$cluster-$name.asok' 2021-05-12T09:34:58.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:34:58.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:34:58.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:34:58.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T09:34:58.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T09:34:58.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:34:58.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:34:58.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:34:58.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:34:58.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:34:58.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/0 2021-05-12T09:34:58.838 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:34:58.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=1159026f-6839-4452-b663-512fe8570944 2021-05-12T09:34:58.839 INFO:tasks.workunit.client.0.smithi028.stdout:add osd0 1159026f-6839-4452-b663-512fe8570944 2021-05-12T09:34:58.839 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 1159026f-6839-4452-b663-512fe8570944' 2021-05-12T09:34:58.840 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:34:58.843 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBCoZtgfSYsMhAAlUBnOsZFeIAh7FnsK2OCRQ== 2021-05-12T09:34:58.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBCoZtgfSYsMhAAlUBnOsZFeIAh7FnsK2OCRQ=="}' 2021-05-12T09:34:58.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 1159026f-6839-4452-b663-512fe8570944 -i td/osd-recovery-prio/0/new.json 2021-05-12T09:34:59.211 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:34:59.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/0/new.json 2021-05-12T09:34:59.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=87db7d12-ce3c-4a9b-8bb7-31c860ffeeb7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185947/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBCoZtgfSYsMhAAlUBnOsZFeIAh7FnsK2OCRQ== --osd-uuid 1159026f-6839-4452-b663-512fe8570944 2021-05-12T09:34:59.247 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:59.244+0000 7fbd765560c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:59.248 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:59.246+0000 7fbd765560c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:59.250 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:59.248+0000 7fbd765560c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:34:59.251 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:59.248+0000 7fbd765560c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T09:34:59.251 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:59.248+0000 7fbd765560c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T09:34:59.252 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:59.248+0000 7fbd765560c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-05-12T09:34:59.291 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:34:59.289+0000 7fbd765560c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2021-05-12T09:35:03.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2021-05-12T09:35:03.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:35:03.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-05-12T09:35:03.876 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd0 key to auth repository 2021-05-12T09:35:03.876 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:35:04.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-05-12T09:35:04.377 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.0 2021-05-12T09:35:04.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=87db7d12-ce3c-4a9b-8bb7-31c860ffeeb7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185947/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:35:04.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:35:04.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:35:04.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:35:04.398 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:04.397+0000 7f9888da90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:35:04.400 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:04.399+0000 7f9888da90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:35:04.402 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:04.400+0000 7f9888da90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:35:04.756 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-05-12T09:35:04.757 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:35:04.757 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=0 2021-05-12T09:35:04.757 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:35:04.757 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:35:04.758 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:35:04.758 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:35:04.758 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:35:04.759 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:35:04.759 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:35:04.920 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:04.919+0000 7f9888da90c0 -1 Falling back to public interface 2021-05-12T09:35:05.160 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:35:06.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:35:06.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:35:06.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:35:06.163 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:35:06.164 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:35:06.164 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:35:06.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:35:07.168 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:07.166+0000 7f9888da90c0 -1 osd.0 0 log_to_monitors {default=true} 2021-05-12T09:35:07.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:35:07.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:35:07.555 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:35:07.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:35:07.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:35:07.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.0 up' 2021-05-12T09:35:07.944 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/187078,v1:127.0.0.1:6803/187078] [v2:127.0.0.1:6804/187078,v1:127.0.0.1:6805/187078] exists,up 1159026f-6839-4452-b663-512fe8570944 2021-05-12T09:35:07.950 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:35:07.950 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:35:07.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:35:07.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-05-12T09:35:07.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:333: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 1 2021-05-12T09:35:07.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-05-12T09:35:07.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-05-12T09:35:07.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-05-12T09:35:07.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-05-12T09:35:07.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2021-05-12T09:35:07.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=87db7d12-ce3c-4a9b-8bb7-31c860ffeeb7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-05-12T09:35:07.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-05-12T09:35:07.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-05-12T09:35:07.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-05-12T09:35:07.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2021-05-12T09:35:07.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2021-05-12T09:35:07.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-05-12T09:35:07.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-05-12T09:35:07.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-05-12T09:35:07.955 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-05-12T09:35:07.955 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-05-12T09:35:07.955 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-05-12T09:35:07.955 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-05-12T09:35:07.955 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:35:07.956 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185947 2021-05-12T09:35:07.956 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.185947/$cluster-$name.asok' 2021-05-12T09:35:07.956 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.185947/$cluster-$name.asok' 2021-05-12T09:35:07.956 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-05-12T09:35:07.957 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-05-12T09:35:07.957 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-05-12T09:35:07.957 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-05-12T09:35:07.957 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-05-12T09:35:07.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-05-12T09:35:07.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-05-12T09:35:07.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:35:07.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-05-12T09:35:07.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-05-12T09:35:07.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-recovery-prio/1 2021-05-12T09:35:07.959 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-05-12T09:35:07.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=3096bbea-98c7-430d-b0d4-0c70f8e7684b 2021-05-12T09:35:07.960 INFO:tasks.workunit.client.0.smithi028.stdout:add osd1 3096bbea-98c7-430d-b0d4-0c70f8e7684b 2021-05-12T09:35:07.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 3096bbea-98c7-430d-b0d4-0c70f8e7684b' 2021-05-12T09:35:07.961 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-05-12T09:35:07.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBLoZtggnc5ORAAUE+WjXFIQIN5p+rZxG6u2Q== 2021-05-12T09:35:07.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBLoZtggnc5ORAAUE+WjXFIQIN5p+rZxG6u2Q=="}' 2021-05-12T09:35:07.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 3096bbea-98c7-430d-b0d4-0c70f8e7684b -i td/osd-recovery-prio/1/new.json 2021-05-12T09:35:08.491 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:35:08.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-recovery-prio/1/new.json 2021-05-12T09:35:08.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=87db7d12-ce3c-4a9b-8bb7-31c860ffeeb7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185947/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBLoZtggnc5ORAAUE+WjXFIQIN5p+rZxG6u2Q== --osd-uuid 3096bbea-98c7-430d-b0d4-0c70f8e7684b 2021-05-12T09:35:08.527 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:08.525+0000 7f953e69c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:35:08.529 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:08.527+0000 7f953e69c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:35:08.530 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:08.528+0000 7f953e69c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:35:08.531 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:08.529+0000 7f953e69c0c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T09:35:08.531 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:08.529+0000 7f953e69c0c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T09:35:08.531 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:08.529+0000 7f953e69c0c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-05-12T09:35:08.662 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:08.660+0000 7f953e69c0c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2021-05-12T09:35:13.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2021-05-12T09:35:13.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-05-12T09:35:13.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-05-12T09:35:13.481 INFO:tasks.workunit.client.0.smithi028.stdout:adding osd1 key to auth repository 2021-05-12T09:35:13.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-05-12T09:35:13.997 INFO:tasks.workunit.client.0.smithi028.stdout:start osd.1 2021-05-12T09:35:13.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-05-12T09:35:13.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-05-12T09:35:13.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=87db7d12-ce3c-4a9b-8bb7-31c860ffeeb7 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.185947/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-05-12T09:35:13.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-05-12T09:35:13.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-05-12T09:35:14.020 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:14.018+0000 7f335239a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:35:14.021 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:14.020+0000 7f335239a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:35:14.023 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:14.021+0000 7f335239a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-05-12T09:35:14.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-05-12T09:35:14.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local state=up 2021-05-12T09:35:14.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: local id=1 2021-05-12T09:35:14.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: status=1 2021-05-12T09:35:14.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i=0 )) 2021-05-12T09:35:14.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:35:14.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 0 2021-05-12T09:35:14.382 INFO:tasks.workunit.client.0.smithi028.stdout:0 2021-05-12T09:35:14.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:35:14.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:35:14.540 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:14.537+0000 7f335239a0c0 -1 Falling back to public interface 2021-05-12T09:35:14.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:35:15.773 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:35:15.773 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:35:15.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 1 2021-05-12T09:35:15.774 INFO:tasks.workunit.client.0.smithi028.stdout:1 2021-05-12T09:35:15.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:35:15.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:35:16.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: sleep 1 2021-05-12T09:35:16.781 INFO:tasks.workunit.client.0.smithi028.stderr:2021-05-12T09:35:16.780+0000 7f335239a0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-05-12T09:35:17.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i++ )) 2021-05-12T09:35:17.187 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: (( i < 300 )) 2021-05-12T09:35:17.187 INFO:tasks.workunit.client.0.smithi028.stdout:2 2021-05-12T09:35:17.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: echo 2 2021-05-12T09:35:17.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: ceph osd dump 2021-05-12T09:35:17.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: grep 'osd.1 up' 2021-05-12T09:35:17.577 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/188463,v1:127.0.0.1:6811/188463] [v2:127.0.0.1:6812/188463,v1:127.0.0.1:6813/188463] exists,up 3096bbea-98c7-430d-b0d4-0c70f8e7684b 2021-05-12T09:35:17.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: status=0 2021-05-12T09:35:17.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: break 2021-05-12T09:35:17.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: return 0 2021-05-12T09:35:17.579 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: seq 1 3 2021-05-12T09:35:17.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T09:35:17.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test1 1 1 2021-05-12T09:35:17.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2021-05-12T09:35:18.229 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'test1' already exists 2021-05-12T09:35:18.238 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:35:19.241 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2021-05-12T09:35:20.060 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 1 size to 2 2021-05-12T09:35:20.069 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T09:35:20.069 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test2 1 1 2021-05-12T09:35:20.070 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2021-05-12T09:35:20.627 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'test2' already exists 2021-05-12T09:35:20.636 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:35:21.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2021-05-12T09:35:22.300 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 2 size to 2 2021-05-12T09:35:22.310 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T09:35:22.310 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test3 1 1 2021-05-12T09:35:22.310 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2021-05-12T09:35:22.946 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'test3' already exists 2021-05-12T09:35:22.955 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-05-12T09:35:23.956 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test3 size 2 2021-05-12T09:35:24.782 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 3 size to 2 2021-05-12T09:35:24.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: sleep 5 2021-05-12T09:35:29.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:343: TEST_recovery_pool_priority: wait_for_clean 2021-05-12T09:35:29.794 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:35:29.794 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:35:29.794 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:35:29.794 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:35:29.795 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T09:35:29.796 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:35:29.796 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:35:29.796 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:35:29.796 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:35:29.797 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:35:29.899 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:35:29.900 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:35:29.900 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:35:29.901 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:35:29.901 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:35:30.284 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:35:30.284 INFO:tasks.workunit.client.0.smithi028.stderr:1' 2021-05-12T09:35:30.284 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:35:30.285 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:35:30.285 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:35:30.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836487 2021-05-12T09:35:30.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836487 2021-05-12T09:35:30.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487' 2021-05-12T09:35:30.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:35:30.419 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:35:30.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672965 2021-05-12T09:35:30.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672965 2021-05-12T09:35:30.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2021-05-12T09:35:30.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:35:30.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-05-12T09:35:30.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:35:30.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:35:30.553 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836487 2021-05-12T09:35:30.554 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:35:30.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836487 2021-05-12T09:35:30.555 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-05-12T09:35:30.555 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836487 2021-05-12T09:35:30.556 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:35:30.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836485 -lt 21474836487 2021-05-12T09:35:30.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:35:31.930 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:35:31.931 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:35:32.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836487 -lt 21474836487 2021-05-12T09:35:32.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:35:32.325 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2021-05-12T09:35:32.325 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:35:32.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:35:32.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672965 2021-05-12T09:35:32.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:35:32.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672965 2021-05-12T09:35:32.328 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672965 2021-05-12T09:35:32.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-05-12T09:35:32.329 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:35:32.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672965 -lt 42949672965 2021-05-12T09:35:32.714 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:35:32.715 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:35:32.715 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:35:33.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 3 == 0 2021-05-12T09:35:33.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:35:33.251 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:35:33.251 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:35:33.251 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:35:33.252 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:35:33.252 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:35:33.252 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:35:33.626 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=3 2021-05-12T09:35:33.626 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:35:33.626 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:35:33.627 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:35:34.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 3 = 3 2021-05-12T09:35:34.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:35:34.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:35:34.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:345: TEST_recovery_pool_priority: ceph pg dump pgs 2021-05-12T09:35:34.479 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:35:34.480 INFO:tasks.workunit.client.0.smithi028.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T09:35:24.961679+0000 0'0 26:16 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:35:22.679565+0000 0'0 2021-05-12T09:35:22.679565+0000 0 2021-05-12T09:35:34.481 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T09:35:20.449264+0000 0'0 25:27 [1,0] 1 [1,0] 1 0'0 2021-05-12T09:35:18.000134+0000 0'0 2021-05-12T09:35:18.000134+0000 0 2021-05-12T09:35:34.481 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-05-12T09:35:22.981445+0000 0'0 25:21 [0,1] 0 [0,1] 0 0'0 2021-05-12T09:35:20.417648+0000 0'0 2021-05-12T09:35:20.417648+0000 0 2021-05-12T09:35:34.482 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:35:34.483 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:35:34.484 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:35:34.493 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:349: TEST_recovery_pool_priority: local PG1 2021-05-12T09:35:34.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:350: TEST_recovery_pool_priority: local POOLNUM1 2021-05-12T09:35:34.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:351: TEST_recovery_pool_priority: local pool1 2021-05-12T09:35:34.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:352: TEST_recovery_pool_priority: local chk_osd1_1 2021-05-12T09:35:34.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:353: TEST_recovery_pool_priority: local chk_osd1_2 2021-05-12T09:35:34.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:355: TEST_recovery_pool_priority: local PG2 2021-05-12T09:35:34.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:356: TEST_recovery_pool_priority: local POOLNUM2 2021-05-12T09:35:34.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:357: TEST_recovery_pool_priority: local pool2 2021-05-12T09:35:34.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:358: TEST_recovery_pool_priority: local chk_osd2_1 2021-05-12T09:35:34.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:359: TEST_recovery_pool_priority: local chk_osd2_2 2021-05-12T09:35:34.496 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: seq 1 3 2021-05-12T09:35:34.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T09:35:34.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: ceph pg map 1.0 --format=json 2021-05-12T09:35:34.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: jq '.acting[]' 2021-05-12T09:35:34.869 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T09:35:34.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: local test_osd1=1 2021-05-12T09:35:34.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T09:35:34.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: local test_osd2=0 2021-05-12T09:35:34.876 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: '[' -z '' ']' 2021-05-12T09:35:34.876 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: PG1=1.0 2021-05-12T09:35:34.877 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: POOLNUM1=1 2021-05-12T09:35:34.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:370: TEST_recovery_pool_priority: pool1=test1 2021-05-12T09:35:34.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:371: TEST_recovery_pool_priority: chk_osd1_1=1 2021-05-12T09:35:34.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:372: TEST_recovery_pool_priority: chk_osd1_2=0 2021-05-12T09:35:34.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T09:35:34.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: ceph pg map 2.0 --format=json 2021-05-12T09:35:34.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: jq '.acting[]' 2021-05-12T09:35:35.257 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2021-05-12T09:35:35.257 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: local test_osd1=0 2021-05-12T09:35:35.258 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2021-05-12T09:35:35.259 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: local test_osd2=1 2021-05-12T09:35:35.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: '[' -z 1.0 ']' 2021-05-12T09:35:35.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:373: TEST_recovery_pool_priority: '[' 1 '!=' 0 ']' 2021-05-12T09:35:35.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:375: TEST_recovery_pool_priority: PG2=2.0 2021-05-12T09:35:35.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:376: TEST_recovery_pool_priority: POOLNUM2=2 2021-05-12T09:35:35.261 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:377: TEST_recovery_pool_priority: pool2=test2 2021-05-12T09:35:35.261 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:378: TEST_recovery_pool_priority: chk_osd2_1=0 2021-05-12T09:35:35.261 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:379: TEST_recovery_pool_priority: chk_osd2_2=1 2021-05-12T09:35:35.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:380: TEST_recovery_pool_priority: break 2021-05-12T09:35:35.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:383: TEST_recovery_pool_priority: rm -f td/osd-recovery-prio/acting 2021-05-12T09:35:35.266 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:385: TEST_recovery_pool_priority: '[' test2 = '' ']' 2021-05-12T09:35:35.267 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: seq 1 3 2021-05-12T09:35:35.268 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T09:35:35.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2021-05-12T09:35:35.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T09:35:35.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2021-05-12T09:35:35.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-05-12T09:35:35.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2021-05-12T09:35:35.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:395: TEST_recovery_pool_priority: delete_pool test3 2021-05-12T09:35:35.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2021-05-12T09:35:35.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2021-05-12T09:35:35.776 INFO:tasks.workunit.client.0.smithi028.stderr:pool 'test3' does not exist 2021-05-12T09:35:35.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:399: TEST_recovery_pool_priority: pool1_extra_prio=1 2021-05-12T09:35:35.786 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:400: TEST_recovery_pool_priority: pool2_extra_prio=2 2021-05-12T09:35:35.788 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:401: TEST_recovery_pool_priority: expr 190 + 1 2021-05-12T09:35:35.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:401: TEST_recovery_pool_priority: pool1_prio=191 2021-05-12T09:35:35.789 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: expr 190 + 2 2021-05-12T09:35:35.789 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: pool2_prio=192 2021-05-12T09:35:35.790 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2021-05-12T09:35:36.472 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 1 size to 1 2021-05-12T09:35:36.482 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: ceph osd pool set test1 recovery_priority 1 2021-05-12T09:35:37.117 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 1 recovery_priority to 1 2021-05-12T09:35:37.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:406: TEST_recovery_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2021-05-12T09:35:37.920 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 2 size to 1 2021-05-12T09:35:37.932 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:407: TEST_recovery_pool_priority: ceph osd pool set test2 recovery_priority 2 2021-05-12T09:35:38.680 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 2 recovery_priority to 2 2021-05-12T09:35:38.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:408: TEST_recovery_pool_priority: wait_for_clean 2021-05-12T09:35:38.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:35:38.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:35:38.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:35:38.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:35:38.693 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T09:35:38.693 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:35:38.694 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:35:38.695 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:35:38.695 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:35:38.695 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:35:38.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:35:38.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:35:38.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:35:38.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:35:38.784 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:35:39.160 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:35:39.160 INFO:tasks.workunit.client.0.smithi028.stderr:1' 2021-05-12T09:35:39.161 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:35:39.161 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:35:39.161 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:35:39.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836491 2021-05-12T09:35:39.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836491 2021-05-12T09:35:39.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491' 2021-05-12T09:35:39.317 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:35:39.318 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:35:39.449 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949672969 2021-05-12T09:35:39.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949672969 2021-05-12T09:35:39.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836491 1-42949672969' 2021-05-12T09:35:39.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:35:39.451 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-05-12T09:35:39.451 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:35:39.452 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:35:39.453 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836491 2021-05-12T09:35:39.453 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:35:39.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836491 2021-05-12T09:35:39.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-05-12T09:35:39.454 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836491 2021-05-12T09:35:39.455 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:35:39.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836491 -lt 21474836491 2021-05-12T09:35:39.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:35:39.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672969 2021-05-12T09:35:39.836 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:35:39.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:35:39.837 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949672969 2021-05-12T09:35:39.838 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:35:39.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949672969 2021-05-12T09:35:39.839 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-05-12T09:35:39.839 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949672969 2021-05-12T09:35:39.839 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:35:40.221 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949672969 -lt 42949672969 2021-05-12T09:35:40.222 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:35:40.222 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:35:40.222 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:35:40.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:35:40.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:35:40.721 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:35:40.721 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:35:40.722 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:35:40.722 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:35:40.722 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:35:40.722 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:35:41.088 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=2 2021-05-12T09:35:41.089 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:35:41.089 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:35:41.089 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:35:41.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 = 2 2021-05-12T09:35:41.591 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: break 2021-05-12T09:35:41.591 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: return 0 2021-05-12T09:35:41.591 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:410: TEST_recovery_pool_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2021-05-12T09:35:41.638 INFO:tasks.workunit.client.0.smithi028.stderr:10+0 records in 2021-05-12T09:35:41.639 INFO:tasks.workunit.client.0.smithi028.stderr:10+0 records out 2021-05-12T09:35:41.639 INFO:tasks.workunit.client.0.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.047182 s, 222 MB/s 2021-05-12T09:35:41.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:411: TEST_recovery_pool_priority: p=1 2021-05-12T09:35:41.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:412: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2021-05-12T09:35:41.640 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: seq 1 200 2021-05-12T09:35:41.640 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:41.641 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2021-05-12T09:35:42.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:42.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2021-05-12T09:35:42.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:42.392 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2021-05-12T09:35:42.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:42.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2021-05-12T09:35:42.994 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:42.995 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2021-05-12T09:35:43.411 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:43.411 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2021-05-12T09:35:43.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:43.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2021-05-12T09:35:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2021-05-12T09:35:44.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:44.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2021-05-12T09:35:45.198 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:45.198 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2021-05-12T09:35:45.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:45.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2021-05-12T09:35:45.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:45.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2021-05-12T09:35:46.243 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:46.243 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2021-05-12T09:35:46.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:46.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2021-05-12T09:35:46.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:46.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2021-05-12T09:35:47.795 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:47.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2021-05-12T09:35:48.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:48.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2021-05-12T09:35:48.505 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:48.506 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2021-05-12T09:35:48.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:48.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2021-05-12T09:35:49.132 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:49.132 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2021-05-12T09:35:49.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:49.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2021-05-12T09:35:49.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:49.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2021-05-12T09:35:50.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:50.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2021-05-12T09:35:51.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:51.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2021-05-12T09:35:51.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:51.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2021-05-12T09:35:52.080 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:52.080 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2021-05-12T09:35:52.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:52.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2021-05-12T09:35:52.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:52.789 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2021-05-12T09:35:53.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:53.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2021-05-12T09:35:54.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:54.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2021-05-12T09:35:54.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:54.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2021-05-12T09:35:54.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:54.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2021-05-12T09:35:54.987 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:54.987 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2021-05-12T09:35:55.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:55.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2021-05-12T09:35:55.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:55.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2021-05-12T09:35:56.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:56.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2021-05-12T09:35:56.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:56.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2021-05-12T09:35:57.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:57.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2021-05-12T09:35:57.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:57.617 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2021-05-12T09:35:58.075 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:58.076 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2021-05-12T09:35:58.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:58.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2021-05-12T09:35:58.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:58.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2021-05-12T09:35:59.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:59.079 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2021-05-12T09:35:59.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:35:59.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2021-05-12T09:36:00.355 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:00.355 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2021-05-12T09:36:00.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:00.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2021-05-12T09:36:00.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:00.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2021-05-12T09:36:01.656 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:01.656 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2021-05-12T09:36:02.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:02.194 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2021-05-12T09:36:02.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:02.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2021-05-12T09:36:03.411 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:03.412 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2021-05-12T09:36:03.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:03.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2021-05-12T09:36:04.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:04.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2021-05-12T09:36:04.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:04.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2021-05-12T09:36:04.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:04.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2021-05-12T09:36:05.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:05.191 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2021-05-12T09:36:05.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:05.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2021-05-12T09:36:06.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:06.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2021-05-12T09:36:06.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:06.753 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2021-05-12T09:36:07.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:07.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2021-05-12T09:36:07.412 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:07.413 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2021-05-12T09:36:07.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:07.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2021-05-12T09:36:08.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:08.148 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2021-05-12T09:36:08.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:08.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2021-05-12T09:36:09.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:09.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2021-05-12T09:36:09.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:09.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2021-05-12T09:36:10.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:10.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2021-05-12T09:36:10.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:10.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2021-05-12T09:36:10.778 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:10.779 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2021-05-12T09:36:11.079 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:11.079 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2021-05-12T09:36:11.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:11.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2021-05-12T09:36:13.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:13.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2021-05-12T09:36:13.374 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:13.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2021-05-12T09:36:13.800 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:13.800 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2021-05-12T09:36:14.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:14.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2021-05-12T09:36:14.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:14.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2021-05-12T09:36:14.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:14.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2021-05-12T09:36:15.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:15.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2021-05-12T09:36:16.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:16.023 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2021-05-12T09:36:16.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:16.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2021-05-12T09:36:16.667 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:16.667 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2021-05-12T09:36:16.974 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:16.975 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2021-05-12T09:36:17.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:17.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2021-05-12T09:36:17.634 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:17.634 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2021-05-12T09:36:18.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:18.644 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2021-05-12T09:36:19.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:19.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2021-05-12T09:36:19.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:19.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2021-05-12T09:36:19.630 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:19.630 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2021-05-12T09:36:20.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:20.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2021-05-12T09:36:20.358 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:20.358 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2021-05-12T09:36:20.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:20.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2021-05-12T09:36:20.975 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:20.975 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2021-05-12T09:36:21.977 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:21.977 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2021-05-12T09:36:22.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:22.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2021-05-12T09:36:22.920 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:22.920 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2021-05-12T09:36:23.597 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:23.597 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2021-05-12T09:36:24.073 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:24.073 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2021-05-12T09:36:24.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:24.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2021-05-12T09:36:25.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:25.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2021-05-12T09:36:25.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:25.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2021-05-12T09:36:25.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:25.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2021-05-12T09:36:26.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:26.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2021-05-12T09:36:26.430 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:26.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2021-05-12T09:36:27.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:27.090 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2021-05-12T09:36:28.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:28.251 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2021-05-12T09:36:28.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:28.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2021-05-12T09:36:28.935 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:28.935 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2021-05-12T09:36:29.265 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:29.265 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2021-05-12T09:36:29.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:29.578 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2021-05-12T09:36:29.996 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:29.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2021-05-12T09:36:30.347 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:30.347 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2021-05-12T09:36:31.165 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:31.165 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2021-05-12T09:36:31.465 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:31.466 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2021-05-12T09:36:31.825 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:31.825 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2021-05-12T09:36:32.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:32.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2021-05-12T09:36:32.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:32.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2021-05-12T09:36:32.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:32.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2021-05-12T09:36:33.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:33.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2021-05-12T09:36:34.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:34.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2021-05-12T09:36:35.448 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:35.448 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2021-05-12T09:36:35.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:35.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2021-05-12T09:36:36.183 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:36.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2021-05-12T09:36:36.493 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:36.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2021-05-12T09:36:37.168 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:37.168 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2021-05-12T09:36:37.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:37.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2021-05-12T09:36:37.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:37.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2021-05-12T09:36:38.179 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:38.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2021-05-12T09:36:38.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:38.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2021-05-12T09:36:38.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:38.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2021-05-12T09:36:39.132 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:39.132 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2021-05-12T09:36:39.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:39.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2021-05-12T09:36:40.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:40.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2021-05-12T09:36:40.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:40.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2021-05-12T09:36:41.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:41.152 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2021-05-12T09:36:41.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:41.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2021-05-12T09:36:41.812 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:41.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2021-05-12T09:36:42.197 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:42.197 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2021-05-12T09:36:42.506 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:42.506 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2021-05-12T09:36:43.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:43.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2021-05-12T09:36:43.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:43.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2021-05-12T09:36:44.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:44.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2021-05-12T09:36:44.977 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:44.977 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2021-05-12T09:36:45.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:45.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2021-05-12T09:36:45.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:45.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2021-05-12T09:36:46.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:46.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2021-05-12T09:36:46.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:46.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2021-05-12T09:36:47.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:47.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2021-05-12T09:36:47.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:47.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2021-05-12T09:36:47.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:47.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2021-05-12T09:36:48.259 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:48.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2021-05-12T09:36:48.577 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:48.577 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2021-05-12T09:36:48.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:48.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2021-05-12T09:36:49.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:49.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2021-05-12T09:36:50.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:50.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2021-05-12T09:36:50.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:50.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2021-05-12T09:36:50.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:50.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2021-05-12T09:36:51.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:51.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2021-05-12T09:36:51.550 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:51.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2021-05-12T09:36:51.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:51.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2021-05-12T09:36:52.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:52.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2021-05-12T09:36:53.211 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:53.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2021-05-12T09:36:53.513 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:53.513 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2021-05-12T09:36:53.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:53.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2021-05-12T09:36:54.214 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:54.215 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2021-05-12T09:36:54.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:54.549 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2021-05-12T09:36:56.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:56.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2021-05-12T09:36:56.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:56.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2021-05-12T09:36:56.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:56.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2021-05-12T09:36:57.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:57.187 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2021-05-12T09:36:57.487 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:57.487 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2021-05-12T09:36:57.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:57.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2021-05-12T09:36:58.241 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:58.241 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2021-05-12T09:36:59.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:59.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2021-05-12T09:36:59.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:59.333 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2021-05-12T09:36:59.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:36:59.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2021-05-12T09:37:00.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:00.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2021-05-12T09:37:00.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:00.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2021-05-12T09:37:00.778 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:00.778 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2021-05-12T09:37:01.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:01.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2021-05-12T09:37:01.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:01.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2021-05-12T09:37:02.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:02.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2021-05-12T09:37:02.640 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:02.640 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2021-05-12T09:37:02.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:02.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2021-05-12T09:37:03.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:03.309 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2021-05-12T09:37:03.600 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:03.601 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2021-05-12T09:37:04.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:04.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2021-05-12T09:37:04.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:04.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2021-05-12T09:37:05.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:05.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2021-05-12T09:37:06.072 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:06.073 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2021-05-12T09:37:06.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:06.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2021-05-12T09:37:06.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:06.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2021-05-12T09:37:07.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:07.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2021-05-12T09:37:07.593 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:07.593 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2021-05-12T09:37:08.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:08.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2021-05-12T09:37:08.854 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:08.854 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2021-05-12T09:37:09.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:09.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2021-05-12T09:37:09.472 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:09.472 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2021-05-12T09:37:09.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:09.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2021-05-12T09:37:10.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:10.233 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2021-05-12T09:37:10.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:10.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2021-05-12T09:37:11.435 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: expr 1 + 1 2021-05-12T09:37:11.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: p=2 2021-05-12T09:37:11.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:412: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2021-05-12T09:37:11.437 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: seq 1 200 2021-05-12T09:37:11.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:11.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj1-p2 td/osd-recovery-prio/data 2021-05-12T09:37:11.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:11.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj2-p2 td/osd-recovery-prio/data 2021-05-12T09:37:12.138 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:12.138 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj3-p2 td/osd-recovery-prio/data 2021-05-12T09:37:12.447 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:12.447 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj4-p2 td/osd-recovery-prio/data 2021-05-12T09:37:12.748 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:12.748 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj5-p2 td/osd-recovery-prio/data 2021-05-12T09:37:13.065 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:13.065 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj6-p2 td/osd-recovery-prio/data 2021-05-12T09:37:13.374 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:13.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj7-p2 td/osd-recovery-prio/data 2021-05-12T09:37:14.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:14.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj8-p2 td/osd-recovery-prio/data 2021-05-12T09:37:14.626 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:14.627 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj9-p2 td/osd-recovery-prio/data 2021-05-12T09:37:14.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:14.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj10-p2 td/osd-recovery-prio/data 2021-05-12T09:37:15.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:15.295 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj11-p2 td/osd-recovery-prio/data 2021-05-12T09:37:15.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:15.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj12-p2 td/osd-recovery-prio/data 2021-05-12T09:37:16.072 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:16.072 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj13-p2 td/osd-recovery-prio/data 2021-05-12T09:37:16.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:16.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj14-p2 td/osd-recovery-prio/data 2021-05-12T09:37:17.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:17.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj15-p2 td/osd-recovery-prio/data 2021-05-12T09:37:17.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:17.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj16-p2 td/osd-recovery-prio/data 2021-05-12T09:37:18.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:18.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj17-p2 td/osd-recovery-prio/data 2021-05-12T09:37:18.309 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:18.309 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj18-p2 td/osd-recovery-prio/data 2021-05-12T09:37:18.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:18.619 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj19-p2 td/osd-recovery-prio/data 2021-05-12T09:37:18.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:18.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj20-p2 td/osd-recovery-prio/data 2021-05-12T09:37:19.261 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:19.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj21-p2 td/osd-recovery-prio/data 2021-05-12T09:37:19.612 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:19.613 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj22-p2 td/osd-recovery-prio/data 2021-05-12T09:37:20.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:20.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj23-p2 td/osd-recovery-prio/data 2021-05-12T09:37:21.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:21.015 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj24-p2 td/osd-recovery-prio/data 2021-05-12T09:37:21.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:21.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj25-p2 td/osd-recovery-prio/data 2021-05-12T09:37:22.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:22.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj26-p2 td/osd-recovery-prio/data 2021-05-12T09:37:22.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:22.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj27-p2 td/osd-recovery-prio/data 2021-05-12T09:37:22.627 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:22.627 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj28-p2 td/osd-recovery-prio/data 2021-05-12T09:37:22.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:22.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj29-p2 td/osd-recovery-prio/data 2021-05-12T09:37:23.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:23.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj30-p2 td/osd-recovery-prio/data 2021-05-12T09:37:24.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:24.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj31-p2 td/osd-recovery-prio/data 2021-05-12T09:37:24.613 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:24.614 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj32-p2 td/osd-recovery-prio/data 2021-05-12T09:37:24.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:24.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj33-p2 td/osd-recovery-prio/data 2021-05-12T09:37:25.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:25.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj34-p2 td/osd-recovery-prio/data 2021-05-12T09:37:25.632 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:25.633 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj35-p2 td/osd-recovery-prio/data 2021-05-12T09:37:26.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:26.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj36-p2 td/osd-recovery-prio/data 2021-05-12T09:37:26.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:26.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj37-p2 td/osd-recovery-prio/data 2021-05-12T09:37:26.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:26.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj38-p2 td/osd-recovery-prio/data 2021-05-12T09:37:27.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:27.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj39-p2 td/osd-recovery-prio/data 2021-05-12T09:37:27.887 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:27.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj40-p2 td/osd-recovery-prio/data 2021-05-12T09:37:28.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:28.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj41-p2 td/osd-recovery-prio/data 2021-05-12T09:37:28.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:28.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj42-p2 td/osd-recovery-prio/data 2021-05-12T09:37:28.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:28.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj43-p2 td/osd-recovery-prio/data 2021-05-12T09:37:29.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:29.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj44-p2 td/osd-recovery-prio/data 2021-05-12T09:37:29.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:29.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj45-p2 td/osd-recovery-prio/data 2021-05-12T09:37:29.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:29.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj46-p2 td/osd-recovery-prio/data 2021-05-12T09:37:30.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:30.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj47-p2 td/osd-recovery-prio/data 2021-05-12T09:37:31.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:31.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj48-p2 td/osd-recovery-prio/data 2021-05-12T09:37:31.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:31.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj49-p2 td/osd-recovery-prio/data 2021-05-12T09:37:32.181 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:32.181 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj50-p2 td/osd-recovery-prio/data 2021-05-12T09:37:32.506 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:32.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj51-p2 td/osd-recovery-prio/data 2021-05-12T09:37:32.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:32.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj52-p2 td/osd-recovery-prio/data 2021-05-12T09:37:33.641 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:33.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj53-p2 td/osd-recovery-prio/data 2021-05-12T09:37:34.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:34.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj54-p2 td/osd-recovery-prio/data 2021-05-12T09:37:34.360 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:34.361 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj55-p2 td/osd-recovery-prio/data 2021-05-12T09:37:34.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:34.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj56-p2 td/osd-recovery-prio/data 2021-05-12T09:37:34.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:34.987 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj57-p2 td/osd-recovery-prio/data 2021-05-12T09:37:35.312 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:35.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj58-p2 td/osd-recovery-prio/data 2021-05-12T09:37:35.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:35.647 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj59-p2 td/osd-recovery-prio/data 2021-05-12T09:37:36.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:36.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj60-p2 td/osd-recovery-prio/data 2021-05-12T09:37:37.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:37.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj61-p2 td/osd-recovery-prio/data 2021-05-12T09:37:37.382 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:37.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj62-p2 td/osd-recovery-prio/data 2021-05-12T09:37:37.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:37.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj63-p2 td/osd-recovery-prio/data 2021-05-12T09:37:38.067 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:38.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj64-p2 td/osd-recovery-prio/data 2021-05-12T09:37:38.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:38.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj65-p2 td/osd-recovery-prio/data 2021-05-12T09:37:38.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:38.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj66-p2 td/osd-recovery-prio/data 2021-05-12T09:37:39.037 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:39.037 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj67-p2 td/osd-recovery-prio/data 2021-05-12T09:37:40.056 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:40.056 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj68-p2 td/osd-recovery-prio/data 2021-05-12T09:37:40.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:40.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj69-p2 td/osd-recovery-prio/data 2021-05-12T09:37:40.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:40.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj70-p2 td/osd-recovery-prio/data 2021-05-12T09:37:41.015 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:41.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj71-p2 td/osd-recovery-prio/data 2021-05-12T09:37:41.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:41.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj72-p2 td/osd-recovery-prio/data 2021-05-12T09:37:42.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:42.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj73-p2 td/osd-recovery-prio/data 2021-05-12T09:37:43.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:43.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj74-p2 td/osd-recovery-prio/data 2021-05-12T09:37:43.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:43.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj75-p2 td/osd-recovery-prio/data 2021-05-12T09:37:43.637 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:43.638 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj76-p2 td/osd-recovery-prio/data 2021-05-12T09:37:44.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:44.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj77-p2 td/osd-recovery-prio/data 2021-05-12T09:37:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj78-p2 td/osd-recovery-prio/data 2021-05-12T09:37:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj79-p2 td/osd-recovery-prio/data 2021-05-12T09:37:44.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:44.991 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj80-p2 td/osd-recovery-prio/data 2021-05-12T09:37:45.309 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:45.309 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj81-p2 td/osd-recovery-prio/data 2021-05-12T09:37:46.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:46.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj82-p2 td/osd-recovery-prio/data 2021-05-12T09:37:46.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:46.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj83-p2 td/osd-recovery-prio/data 2021-05-12T09:37:46.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:46.987 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj84-p2 td/osd-recovery-prio/data 2021-05-12T09:37:47.312 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:47.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj85-p2 td/osd-recovery-prio/data 2021-05-12T09:37:47.666 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:47.667 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj86-p2 td/osd-recovery-prio/data 2021-05-12T09:37:48.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:48.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj87-p2 td/osd-recovery-prio/data 2021-05-12T09:37:48.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:48.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj88-p2 td/osd-recovery-prio/data 2021-05-12T09:37:49.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:49.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj89-p2 td/osd-recovery-prio/data 2021-05-12T09:37:49.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:49.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj90-p2 td/osd-recovery-prio/data 2021-05-12T09:37:49.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:49.894 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj91-p2 td/osd-recovery-prio/data 2021-05-12T09:37:50.285 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:50.286 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj92-p2 td/osd-recovery-prio/data 2021-05-12T09:37:50.586 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:50.586 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj93-p2 td/osd-recovery-prio/data 2021-05-12T09:37:50.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:50.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj94-p2 td/osd-recovery-prio/data 2021-05-12T09:37:51.568 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:51.568 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj95-p2 td/osd-recovery-prio/data 2021-05-12T09:37:53.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:53.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj96-p2 td/osd-recovery-prio/data 2021-05-12T09:37:54.134 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:54.134 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj97-p2 td/osd-recovery-prio/data 2021-05-12T09:37:54.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:54.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj98-p2 td/osd-recovery-prio/data 2021-05-12T09:37:54.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:54.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj99-p2 td/osd-recovery-prio/data 2021-05-12T09:37:55.303 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:55.303 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj100-p2 td/osd-recovery-prio/data 2021-05-12T09:37:55.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:55.612 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj101-p2 td/osd-recovery-prio/data 2021-05-12T09:37:55.987 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:55.988 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj102-p2 td/osd-recovery-prio/data 2021-05-12T09:37:56.447 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:56.448 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj103-p2 td/osd-recovery-prio/data 2021-05-12T09:37:57.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:57.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj104-p2 td/osd-recovery-prio/data 2021-05-12T09:37:57.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:57.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj105-p2 td/osd-recovery-prio/data 2021-05-12T09:37:58.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:58.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj106-p2 td/osd-recovery-prio/data 2021-05-12T09:37:58.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:58.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj107-p2 td/osd-recovery-prio/data 2021-05-12T09:37:59.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:59.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj108-p2 td/osd-recovery-prio/data 2021-05-12T09:37:59.489 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:59.489 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj109-p2 td/osd-recovery-prio/data 2021-05-12T09:37:59.864 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:37:59.865 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj110-p2 td/osd-recovery-prio/data 2021-05-12T09:38:00.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:00.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj111-p2 td/osd-recovery-prio/data 2021-05-12T09:38:00.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:00.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj112-p2 td/osd-recovery-prio/data 2021-05-12T09:38:00.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:00.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj113-p2 td/osd-recovery-prio/data 2021-05-12T09:38:01.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:01.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj114-p2 td/osd-recovery-prio/data 2021-05-12T09:38:02.211 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:02.211 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj115-p2 td/osd-recovery-prio/data 2021-05-12T09:38:02.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:02.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj116-p2 td/osd-recovery-prio/data 2021-05-12T09:38:02.863 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:02.863 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj117-p2 td/osd-recovery-prio/data 2021-05-12T09:38:03.197 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:03.198 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj118-p2 td/osd-recovery-prio/data 2021-05-12T09:38:04.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:04.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj119-p2 td/osd-recovery-prio/data 2021-05-12T09:38:04.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:04.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj120-p2 td/osd-recovery-prio/data 2021-05-12T09:38:05.267 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:05.267 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj121-p2 td/osd-recovery-prio/data 2021-05-12T09:38:05.568 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:05.568 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj122-p2 td/osd-recovery-prio/data 2021-05-12T09:38:05.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:05.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj123-p2 td/osd-recovery-prio/data 2021-05-12T09:38:06.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:06.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj124-p2 td/osd-recovery-prio/data 2021-05-12T09:38:06.629 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:06.629 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj125-p2 td/osd-recovery-prio/data 2021-05-12T09:38:07.613 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:07.614 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj126-p2 td/osd-recovery-prio/data 2021-05-12T09:38:08.081 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:08.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj127-p2 td/osd-recovery-prio/data 2021-05-12T09:38:08.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:08.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj128-p2 td/osd-recovery-prio/data 2021-05-12T09:38:08.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:08.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj129-p2 td/osd-recovery-prio/data 2021-05-12T09:38:09.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:09.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj130-p2 td/osd-recovery-prio/data 2021-05-12T09:38:09.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:09.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj131-p2 td/osd-recovery-prio/data 2021-05-12T09:38:09.668 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:09.669 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj132-p2 td/osd-recovery-prio/data 2021-05-12T09:38:10.094 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:10.095 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj133-p2 td/osd-recovery-prio/data 2021-05-12T09:38:10.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:10.930 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj134-p2 td/osd-recovery-prio/data 2021-05-12T09:38:11.229 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:11.229 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj135-p2 td/osd-recovery-prio/data 2021-05-12T09:38:11.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:11.564 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj136-p2 td/osd-recovery-prio/data 2021-05-12T09:38:12.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:12.057 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj137-p2 td/osd-recovery-prio/data 2021-05-12T09:38:12.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:12.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj138-p2 td/osd-recovery-prio/data 2021-05-12T09:38:12.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:12.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj139-p2 td/osd-recovery-prio/data 2021-05-12T09:38:13.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:13.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj140-p2 td/osd-recovery-prio/data 2021-05-12T09:38:13.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:13.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj141-p2 td/osd-recovery-prio/data 2021-05-12T09:38:14.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:14.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj142-p2 td/osd-recovery-prio/data 2021-05-12T09:38:14.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:14.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj143-p2 td/osd-recovery-prio/data 2021-05-12T09:38:15.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:15.122 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj144-p2 td/osd-recovery-prio/data 2021-05-12T09:38:15.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:15.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj145-p2 td/osd-recovery-prio/data 2021-05-12T09:38:15.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:15.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj146-p2 td/osd-recovery-prio/data 2021-05-12T09:38:16.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:16.157 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj147-p2 td/osd-recovery-prio/data 2021-05-12T09:38:17.042 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:17.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj148-p2 td/osd-recovery-prio/data 2021-05-12T09:38:17.351 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:17.351 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj149-p2 td/osd-recovery-prio/data 2021-05-12T09:38:17.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:17.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj150-p2 td/osd-recovery-prio/data 2021-05-12T09:38:18.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:18.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj151-p2 td/osd-recovery-prio/data 2021-05-12T09:38:18.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:18.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj152-p2 td/osd-recovery-prio/data 2021-05-12T09:38:18.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:18.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj153-p2 td/osd-recovery-prio/data 2021-05-12T09:38:19.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:19.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj154-p2 td/osd-recovery-prio/data 2021-05-12T09:38:19.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:19.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj155-p2 td/osd-recovery-prio/data 2021-05-12T09:38:20.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:20.342 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj156-p2 td/osd-recovery-prio/data 2021-05-12T09:38:20.666 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:20.690 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj157-p2 td/osd-recovery-prio/data 2021-05-12T09:38:21.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:21.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj158-p2 td/osd-recovery-prio/data 2021-05-12T09:38:21.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:21.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj159-p2 td/osd-recovery-prio/data 2021-05-12T09:38:21.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:21.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj160-p2 td/osd-recovery-prio/data 2021-05-12T09:38:22.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:22.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj161-p2 td/osd-recovery-prio/data 2021-05-12T09:38:22.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:22.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj162-p2 td/osd-recovery-prio/data 2021-05-12T09:38:23.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:23.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj163-p2 td/osd-recovery-prio/data 2021-05-12T09:38:23.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:23.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj164-p2 td/osd-recovery-prio/data 2021-05-12T09:38:23.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:23.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj165-p2 td/osd-recovery-prio/data 2021-05-12T09:38:24.557 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:24.558 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj166-p2 td/osd-recovery-prio/data 2021-05-12T09:38:24.858 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:24.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj167-p2 td/osd-recovery-prio/data 2021-05-12T09:38:25.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:25.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj168-p2 td/osd-recovery-prio/data 2021-05-12T09:38:25.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:25.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj169-p2 td/osd-recovery-prio/data 2021-05-12T09:38:26.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:26.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj170-p2 td/osd-recovery-prio/data 2021-05-12T09:38:26.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:26.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj171-p2 td/osd-recovery-prio/data 2021-05-12T09:38:27.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:27.155 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj172-p2 td/osd-recovery-prio/data 2021-05-12T09:38:27.455 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:27.455 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj173-p2 td/osd-recovery-prio/data 2021-05-12T09:38:27.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:27.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj174-p2 td/osd-recovery-prio/data 2021-05-12T09:38:28.182 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:28.183 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj175-p2 td/osd-recovery-prio/data 2021-05-12T09:38:28.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:28.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj176-p2 td/osd-recovery-prio/data 2021-05-12T09:38:28.826 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:28.826 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj177-p2 td/osd-recovery-prio/data 2021-05-12T09:38:29.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:29.685 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj178-p2 td/osd-recovery-prio/data 2021-05-12T09:38:30.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:30.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj179-p2 td/osd-recovery-prio/data 2021-05-12T09:38:30.403 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:30.403 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj180-p2 td/osd-recovery-prio/data 2021-05-12T09:38:30.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:30.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj181-p2 td/osd-recovery-prio/data 2021-05-12T09:38:31.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:31.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj182-p2 td/osd-recovery-prio/data 2021-05-12T09:38:31.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:31.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj183-p2 td/osd-recovery-prio/data 2021-05-12T09:38:31.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:31.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj184-p2 td/osd-recovery-prio/data 2021-05-12T09:38:32.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:32.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj185-p2 td/osd-recovery-prio/data 2021-05-12T09:38:33.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:33.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj186-p2 td/osd-recovery-prio/data 2021-05-12T09:38:33.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:33.360 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj187-p2 td/osd-recovery-prio/data 2021-05-12T09:38:33.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:33.702 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj188-p2 td/osd-recovery-prio/data 2021-05-12T09:38:34.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:34.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj189-p2 td/osd-recovery-prio/data 2021-05-12T09:38:35.165 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:35.166 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj190-p2 td/osd-recovery-prio/data 2021-05-12T09:38:36.172 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:36.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj191-p2 td/osd-recovery-prio/data 2021-05-12T09:38:36.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:36.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj192-p2 td/osd-recovery-prio/data 2021-05-12T09:38:36.807 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:36.808 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj193-p2 td/osd-recovery-prio/data 2021-05-12T09:38:37.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:37.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj194-p2 td/osd-recovery-prio/data 2021-05-12T09:38:37.667 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:37.668 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj195-p2 td/osd-recovery-prio/data 2021-05-12T09:38:38.093 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:38.094 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj196-p2 td/osd-recovery-prio/data 2021-05-12T09:38:38.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:38.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj197-p2 td/osd-recovery-prio/data 2021-05-12T09:38:39.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:39.139 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj198-p2 td/osd-recovery-prio/data 2021-05-12T09:38:39.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:39.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj199-p2 td/osd-recovery-prio/data 2021-05-12T09:38:39.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-05-12T09:38:39.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj200-p2 td/osd-recovery-prio/data 2021-05-12T09:38:40.174 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: expr 2 + 1 2021-05-12T09:38:40.175 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: p=3 2021-05-12T09:38:40.177 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: get_not_primary test1 obj1-p1 2021-05-12T09:38:40.177 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local poolname=test1 2021-05-12T09:38:40.177 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local objectname=obj1-p1 2021-05-12T09:38:40.177 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: get_primary test1 obj1-p1 2021-05-12T09:38:40.178 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local poolname=test1 2021-05-12T09:38:40.178 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: local objectname=obj1-p1 2021-05-12T09:38:40.178 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: ceph --format json osd map test1 obj1-p1 2021-05-12T09:38:40.178 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: jq .acting_primary 2021-05-12T09:38:40.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local primary=1 2021-05-12T09:38:40.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: ceph --format json osd map test1 obj1-p1 2021-05-12T09:38:40.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1173: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-05-12T09:38:40.976 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: local otherosd=null 2021-05-12T09:38:40.976 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:423: TEST_recovery_pool_priority: ceph pg dump pgs 2021-05-12T09:38:41.354 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:38:41.354 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-05-12T09:35:36.643600+0000 36'600 36:642 [1] 1 [1] 1 0'0 2021-05-12T09:35:18.000134+0000 0'0 2021-05-12T09:35:18.000134+0000 0 2021-05-12T09:38:41.355 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 196 0 0 0 0 2048917504 0 0 586 586 active+clean 2021-05-12T09:35:38.274321+0000 36'586 36:622 [0] 0 [0] 0 0'0 2021-05-12T09:35:20.417648+0000 0'0 2021-05-12T09:35:20.417648+0000 0 2021-05-12T09:38:41.355 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:38:41.355 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:38:41.356 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:38:41.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: ERRORS=0 2021-05-12T09:38:41.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:426: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2021-05-12T09:38:42.017 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 1 size to 2 2021-05-12T09:38:42.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:427: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2021-05-12T09:38:42.753 INFO:tasks.workunit.client.0.smithi028.stderr:set pool 2 size to 2 2021-05-12T09:38:42.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:430: TEST_recovery_pool_priority: ceph pg dump pgs 2021-05-12T09:38:43.166 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:38:43.166 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-05-12T09:35:36.643600+0000 36'600 36:642 [1] 1 [1] 1 0'0 2021-05-12T09:35:18.000134+0000 0'0 2021-05-12T09:35:18.000134+0000 0 2021-05-12T09:38:43.166 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 196 0 0 0 0 2048917504 0 0 586 586 active+clean 2021-05-12T09:35:38.274321+0000 36'586 36:622 [0] 0 [0] 0 0'0 2021-05-12T09:35:20.417648+0000 0'0 2021-05-12T09:35:20.417648+0000 0 2021-05-12T09:38:43.166 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:38:43.167 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:38:43.168 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:38:43.177 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:433: TEST_recovery_pool_priority: set -o pipefail 2021-05-12T09:38:43.178 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:434: TEST_recovery_pool_priority: count=0 2021-05-12T09:38:43.178 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-05-12T09:38:43.179 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-05-12T09:38:43.179 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-05-12T09:38:43.179 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-05-12T09:38:43.597 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:38:43.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 0 == 2 2021-05-12T09:38:43.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: sleep 2 2021-05-12T09:38:45.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: test 0 -eq 10 2021-05-12T09:38:45.608 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: expr 0 + 1 2021-05-12T09:38:45.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: count=1 2021-05-12T09:38:45.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-05-12T09:38:45.611 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-05-12T09:38:45.611 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-05-12T09:38:45.611 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-05-12T09:38:45.973 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:38:45.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 0 == 2 2021-05-12T09:38:45.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: sleep 2 2021-05-12T09:38:47.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: test 1 -eq 10 2021-05-12T09:38:47.984 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: expr 1 + 1 2021-05-12T09:38:47.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: count=2 2021-05-12T09:38:47.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-05-12T09:38:47.987 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-05-12T09:38:47.987 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-05-12T09:38:47.987 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-05-12T09:38:48.353 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:38:48.362 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 0 == 2 2021-05-12T09:38:48.363 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: sleep 2 2021-05-12T09:38:50.364 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: test 2 -eq 10 2021-05-12T09:38:50.365 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: expr 2 + 1 2021-05-12T09:38:50.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: count=3 2021-05-12T09:38:50.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-05-12T09:38:50.367 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-05-12T09:38:50.368 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-05-12T09:38:50.368 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-05-12T09:38:50.725 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:38:50.734 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 2 == 2 2021-05-12T09:38:50.734 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:440: TEST_recovery_pool_priority: break 2021-05-12T09:38:50.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:450: TEST_recovery_pool_priority: set +o pipefail 2021-05-12T09:38:50.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: ceph pg dump pgs 2021-05-12T09:38:51.097 INFO:tasks.workunit.client.0.smithi028.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-05-12T09:38:51.097 INFO:tasks.workunit.client.0.smithi028.stdout:1.0 200 0 199 0 0 2097152000 0 0 600 600 active+recovering+undersized+degraded+remapped 2021-05-12T09:38:43.560899+0000 36'600 42:656 [1,0] 1 [1] 1 0'0 2021-05-12T09:35:18.000134+0000 0'0 2021-05-12T09:35:18.000134+0000 0 2021-05-12T09:38:51.098 INFO:tasks.workunit.client.0.smithi028.stdout:2.0 200 0 199 0 0 2097152000 0 0 600 600 active+recovering+undersized+degraded+remapped 2021-05-12T09:38:43.657311+0000 36'600 42:651 [0,1] 0 [0] 0 0'0 2021-05-12T09:35:20.417648+0000 0'0 2021-05-12T09:35:20.417648+0000 0 2021-05-12T09:38:51.098 INFO:tasks.workunit.client.0.smithi028.stdout: 2021-05-12T09:38:51.098 INFO:tasks.workunit.client.0.smithi028.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-05-12T09:38:51.099 INFO:tasks.workunit.client.0.smithi028.stderr:dumped pgs 2021-05-12T09:38:51.110 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: get_asok_path osd.1 2021-05-12T09:38:51.110 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-05-12T09:38:51.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-05-12T09:38:51.111 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:38:51.111 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:38:51.111 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185947 2021-05-12T09:38:51.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185947/ceph-osd.1.asok 2021-05-12T09:38:51.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: CEPH_ARGS= 2021-05-12T09:38:51.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.185947/ceph-osd.1.asok dump_recovery_reservations 2021-05-12T09:38:51.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:454: TEST_recovery_pool_priority: echo osd.1 2021-05-12T09:38:51.216 INFO:tasks.workunit.client.0.smithi028.stdout:osd.1 2021-05-12T09:38:51.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:455: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T09:38:51.217 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:38:51.218 INFO:tasks.workunit.client.0.smithi028.stdout: "local_reservations": { 2021-05-12T09:38:51.218 INFO:tasks.workunit.client.0.smithi028.stdout: "max_allowed": 1000, 2021-05-12T09:38:51.218 INFO:tasks.workunit.client.0.smithi028.stdout: "min_priority": 0, 2021-05-12T09:38:51.218 INFO:tasks.workunit.client.0.smithi028.stdout: "queues": [], 2021-05-12T09:38:51.218 INFO:tasks.workunit.client.0.smithi028.stdout: "in_progress": [ 2021-05-12T09:38:51.219 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:38:51.219 INFO:tasks.workunit.client.0.smithi028.stdout: "item": "1.0", 2021-05-12T09:38:51.219 INFO:tasks.workunit.client.0.smithi028.stdout: "prio": 191, 2021-05-12T09:38:51.219 INFO:tasks.workunit.client.0.smithi028.stdout: "can_preempt": true 2021-05-12T09:38:51.220 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:38:51.220 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:38:51.220 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:38:51.221 INFO:tasks.workunit.client.0.smithi028.stdout: "remote_reservations": { 2021-05-12T09:38:51.221 INFO:tasks.workunit.client.0.smithi028.stdout: "max_allowed": 1000, 2021-05-12T09:38:51.221 INFO:tasks.workunit.client.0.smithi028.stdout: "min_priority": 0, 2021-05-12T09:38:51.221 INFO:tasks.workunit.client.0.smithi028.stdout: "queues": [], 2021-05-12T09:38:51.222 INFO:tasks.workunit.client.0.smithi028.stdout: "in_progress": [ 2021-05-12T09:38:51.222 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:38:51.222 INFO:tasks.workunit.client.0.smithi028.stdout: "item": "2.0", 2021-05-12T09:38:51.223 INFO:tasks.workunit.client.0.smithi028.stdout: "prio": 192, 2021-05-12T09:38:51.223 INFO:tasks.workunit.client.0.smithi028.stdout: "can_preempt": true 2021-05-12T09:38:51.223 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:38:51.223 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:38:51.224 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:38:51.224 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:38:51.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: get_asok_path osd.0 2021-05-12T09:38:51.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-05-12T09:38:51.226 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-05-12T09:38:51.226 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-05-12T09:38:51.226 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-05-12T09:38:51.226 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.185947 2021-05-12T09:38:51.227 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.185947/ceph-osd.0.asok 2021-05-12T09:38:51.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: CEPH_ARGS= 2021-05-12T09:38:51.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.185947/ceph-osd.0.asok dump_recovery_reservations 2021-05-12T09:38:51.322 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:457: TEST_recovery_pool_priority: echo osd.0 2021-05-12T09:38:51.323 INFO:tasks.workunit.client.0.smithi028.stdout:osd.0 2021-05-12T09:38:51.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T09:38:51.325 INFO:tasks.workunit.client.0.smithi028.stdout:{ 2021-05-12T09:38:51.325 INFO:tasks.workunit.client.0.smithi028.stdout: "local_reservations": { 2021-05-12T09:38:51.325 INFO:tasks.workunit.client.0.smithi028.stdout: "max_allowed": 1000, 2021-05-12T09:38:51.326 INFO:tasks.workunit.client.0.smithi028.stdout: "min_priority": 0, 2021-05-12T09:38:51.326 INFO:tasks.workunit.client.0.smithi028.stdout: "queues": [], 2021-05-12T09:38:51.326 INFO:tasks.workunit.client.0.smithi028.stdout: "in_progress": [ 2021-05-12T09:38:51.326 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:38:51.326 INFO:tasks.workunit.client.0.smithi028.stdout: "item": "2.0", 2021-05-12T09:38:51.327 INFO:tasks.workunit.client.0.smithi028.stdout: "prio": 192, 2021-05-12T09:38:51.327 INFO:tasks.workunit.client.0.smithi028.stdout: "can_preempt": true 2021-05-12T09:38:51.327 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:38:51.327 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:38:51.327 INFO:tasks.workunit.client.0.smithi028.stdout: }, 2021-05-12T09:38:51.328 INFO:tasks.workunit.client.0.smithi028.stdout: "remote_reservations": { 2021-05-12T09:38:51.328 INFO:tasks.workunit.client.0.smithi028.stdout: "max_allowed": 1000, 2021-05-12T09:38:51.328 INFO:tasks.workunit.client.0.smithi028.stdout: "min_priority": 0, 2021-05-12T09:38:51.328 INFO:tasks.workunit.client.0.smithi028.stdout: "queues": [], 2021-05-12T09:38:51.329 INFO:tasks.workunit.client.0.smithi028.stdout: "in_progress": [ 2021-05-12T09:38:51.329 INFO:tasks.workunit.client.0.smithi028.stdout: { 2021-05-12T09:38:51.329 INFO:tasks.workunit.client.0.smithi028.stdout: "item": "1.0", 2021-05-12T09:38:51.329 INFO:tasks.workunit.client.0.smithi028.stdout: "prio": 191, 2021-05-12T09:38:51.330 INFO:tasks.workunit.client.0.smithi028.stdout: "can_preempt": true 2021-05-12T09:38:51.330 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:38:51.330 INFO:tasks.workunit.client.0.smithi028.stdout: ] 2021-05-12T09:38:51.330 INFO:tasks.workunit.client.0.smithi028.stdout: } 2021-05-12T09:38:51.330 INFO:tasks.workunit.client.0.smithi028.stdout:} 2021-05-12T09:38:51.332 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T09:38:51.332 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2021-05-12T09:38:51.333 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2021-05-12T09:38:51.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: ITEM=1.0 2021-05-12T09:38:51.333 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:462: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2021-05-12T09:38:51.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T09:38:51.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2021-05-12T09:38:51.334 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: PRIO=191 2021-05-12T09:38:51.334 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:468: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2021-05-12T09:38:51.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T09:38:51.335 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2021-05-12T09:38:51.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2021-05-12T09:38:51.336 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: ITEM=1.0 2021-05-12T09:38:51.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:477: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2021-05-12T09:38:51.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T09:38:51.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2021-05-12T09:38:51.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: PRIO=191 2021-05-12T09:38:51.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:483: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2021-05-12T09:38:51.339 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T09:38:51.339 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2021-05-12T09:38:51.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2021-05-12T09:38:51.340 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: ITEM=2.0 2021-05-12T09:38:51.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:492: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2021-05-12T09:38:51.341 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:497: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-05-12T09:38:51.341 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:497: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2021-05-12T09:38:51.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:497: TEST_recovery_pool_priority: PRIO=192 2021-05-12T09:38:51.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:498: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2021-05-12T09:38:51.344 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T09:38:51.345 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2021-05-12T09:38:51.347 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2021-05-12T09:38:51.347 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: ITEM=2.0 2021-05-12T09:38:51.348 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:507: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2021-05-12T09:38:51.349 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:512: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-05-12T09:38:51.349 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:512: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2021-05-12T09:38:51.351 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:512: TEST_recovery_pool_priority: PRIO=192 2021-05-12T09:38:51.351 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:513: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2021-05-12T09:38:51.351 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:520: TEST_recovery_pool_priority: wait_for_clean 2021-05-12T09:38:51.351 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cmd= 2021-05-12T09:38:51.352 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local num_active_clean=-1 2021-05-12T09:38:51.352 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local cur_active_clean 2021-05-12T09:38:51.352 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-05-12T09:38:51.353 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: get_timeout_delays 90 .1 2021-05-12T09:38:51.354 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -q -o xtrace 2021-05-12T09:38:51.354 INFO:tasks.workunit.client.0.smithi028.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: echo true 2021-05-12T09:38:51.354 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: local trace=true 2021-05-12T09:38:51.355 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: true 2021-05-12T09:38:51.355 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_timeout_delays: shopt -u -o xtrace 2021-05-12T09:38:51.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -a delays 2021-05-12T09:38:51.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local -i loop=0 2021-05-12T09:38:51.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: flush_pg_stats 2021-05-12T09:38:51.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: local timeout=300 2021-05-12T09:38:51.445 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ceph osd ls 2021-05-12T09:38:51.845 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: ids='0 2021-05-12T09:38:51.846 INFO:tasks.workunit.client.0.smithi028.stderr:1' 2021-05-12T09:38:51.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seqs= 2021-05-12T09:38:51.846 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:38:51.846 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-05-12T09:38:51.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=21474836531 2021-05-12T09:38:51.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 21474836531 2021-05-12T09:38:51.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836531' 2021-05-12T09:38:51.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: for osd in $ids 2021-05-12T09:38:51.980 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-05-12T09:38:52.111 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: seq=42949673009 2021-05-12T09:38:52.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: test -z 42949673009 2021-05-12T09:38:52.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: seqs=' 0-21474836531 1-42949673009' 2021-05-12T09:38:52.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:38:52.113 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836531 2021-05-12T09:38:52.113 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:38:52.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=0 2021-05-12T09:38:52.114 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 0-21474836531 2021-05-12T09:38:52.115 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:38:52.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=21474836531 2021-05-12T09:38:52.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2021-05-12T09:38:52.116 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.0 seq 21474836531 2021-05-12T09:38:52.116 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:38:52.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836529 -lt 21474836531 2021-05-12T09:38:52.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: sleep 1 2021-05-12T09:38:53.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: '[' 300 -eq 0 ']' 2021-05-12T09:38:53.542 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 0 2021-05-12T09:38:53.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 21474836532 -lt 21474836531 2021-05-12T09:38:53.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: for s in $seqs 2021-05-12T09:38:53.927 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673009 2021-05-12T09:38:53.927 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 1 2021-05-12T09:38:53.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: osd=1 2021-05-12T09:38:53.930 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 1-42949673009 2021-05-12T09:38:53.930 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: cut -d - -f 2 2021-05-12T09:38:53.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: seq=42949673009 2021-05-12T09:38:53.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: echo 'waiting osd.1 seq 42949673009' 2021-05-12T09:38:53.931 INFO:tasks.workunit.client.0.smithi028.stdout:waiting osd.1 seq 42949673009 2021-05-12T09:38:53.932 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: ceph osd last-stat-seq 1 2021-05-12T09:38:54.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: test 42949673010 -lt 42949673009 2021-05-12T09:38:54.314 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: get_num_pgs 2021-05-12T09:38:54.315 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:38:54.315 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:38:54.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: test 2 == 0 2021-05-12T09:38:54.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:38:54.819 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:38:54.819 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:38:54.820 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:38:54.820 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:38:54.820 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:38:54.820 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:38:55.189 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:38:55.190 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:38:55.190 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:38:55.190 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:38:55.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:38:55.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' -1 2021-05-12T09:38:55.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: loop=0 2021-05-12T09:38:55.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: num_active_clean=0 2021-05-12T09:38:55.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:38:55.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:38:55.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:38:55.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:38:55.800 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:38:55.800 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:38:55.800 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:38:55.800 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:38:55.801 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:38:55.801 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:38:56.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:38:56.174 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:38:56.175 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:38:56.175 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:38:56.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:38:56.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:38:56.679 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:38:56.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:38:56.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:38:56.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:38:56.680 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:38:56.681 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:38:56.681 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:38:57.187 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368372 2021-05-12T09:38:57.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368372 '!=' null 2021-05-12T09:38:57.188 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:38:57.189 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:38:57.189 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:38:57.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:38:57.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:38:57.290 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:38:57.291 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:38:57.291 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:38:57.292 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:38:57.292 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:38:57.292 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:38:57.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:38:57.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:38:57.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:38:57.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:38:58.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:38:58.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:38:58.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:38:58.251 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:38:58.251 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:38:58.252 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:38:58.252 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:38:58.253 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:38:58.253 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:38:58.753 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368386 2021-05-12T09:38:58.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368386 '!=' null 2021-05-12T09:38:58.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:38:58.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:38:58.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:38:58.854 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:38:58.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:38:58.856 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:38:58.856 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:38:58.856 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:38:58.856 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:38:58.857 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:38:58.857 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:38:59.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:38:59.235 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:38:59.236 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:38:59.236 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:38:59.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:38:59.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:38:59.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:38:59.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:38:59.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:38:59.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:38:59.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:38:59.741 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:38:59.742 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:00.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989404 2021-05-12T09:39:00.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989404 '!=' null 2021-05-12T09:39:00.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:00.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:00.246 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:00.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:00.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:00.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:00.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:00.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:00.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:00.347 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:00.347 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:00.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:00.729 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:00.730 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:00.730 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:01.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:01.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:01.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:01.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:01.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:01.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:01.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:01.226 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:01.226 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:01.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989404 2021-05-12T09:39:01.736 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989404 '!=' null 2021-05-12T09:39:01.736 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:01.736 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:01.737 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:01.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:01.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:01.837 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:01.838 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:01.838 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:01.838 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:01.838 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:01.839 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:02.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:02.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:02.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:02.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:02.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:02.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:02.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:02.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:02.753 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:02.753 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:02.753 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:02.754 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:02.754 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:03.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242038 2021-05-12T09:39:03.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242038 '!=' null 2021-05-12T09:39:03.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:03.257 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:03.257 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:03.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:03.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:03.358 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:03.358 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:03.358 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:03.358 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:03.359 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:03.359 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:03.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:03.728 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:03.729 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:03.729 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:04.238 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:04.238 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:04.238 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:04.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:04.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:04.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:04.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:04.240 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:04.240 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:04.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863074 2021-05-12T09:39:04.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863074 '!=' null 2021-05-12T09:39:04.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:04.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:04.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:04.842 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:04.843 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:04.844 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:04.844 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:04.844 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:04.845 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:04.845 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:04.845 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:05.213 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:05.214 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:05.214 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:05.215 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:05.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:05.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:05.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:05.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:05.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:05.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:05.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:05.724 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:05.724 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:06.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242081 2021-05-12T09:39:06.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242081 '!=' null 2021-05-12T09:39:06.238 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:06.238 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:06.238 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:06.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:06.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:06.342 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:06.343 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:06.343 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:06.343 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:06.343 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:06.344 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:06.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:06.725 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:06.726 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:06.726 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:07.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:07.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:07.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:07.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:07.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:07.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:07.233 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:07.233 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:07.234 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:07.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242081 2021-05-12T09:39:07.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242081 '!=' null 2021-05-12T09:39:07.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:07.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:07.734 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:07.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:07.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:07.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:07.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:07.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:07.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:07.836 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:07.836 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:08.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:08.206 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:08.206 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:08.207 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:08.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:08.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:08.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:08.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:08.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:08.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:08.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:08.713 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:08.714 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:09.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242058 2021-05-12T09:39:09.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242058 '!=' null 2021-05-12T09:39:09.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:09.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:09.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:09.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:09.326 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:09.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:09.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:09.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:09.328 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:09.328 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:09.328 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:09.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:09.700 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:09.700 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:09.701 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:10.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:10.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:10.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:10.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:10.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:10.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:10.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:10.220 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:10.220 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:10.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989395 2021-05-12T09:39:10.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989395 '!=' null 2021-05-12T09:39:10.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:10.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:10.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:10.825 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:10.826 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:10.827 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:10.827 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:10.828 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:10.828 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:10.828 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:10.828 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:11.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:11.204 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:11.204 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:11.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:11.708 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:11.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:11.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:11.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:11.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:11.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:11.710 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:11.711 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:11.711 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:12.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368374 2021-05-12T09:39:12.312 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368374 '!=' null 2021-05-12T09:39:12.312 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:12.312 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:12.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:12.411 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:12.412 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:12.413 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:12.413 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:12.413 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:12.414 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:12.414 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:12.414 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:12.790 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:12.791 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:12.792 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:12.792 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:13.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:13.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:13.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:13.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:13.290 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:13.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:13.291 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:13.292 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:13.292 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:13.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368374 2021-05-12T09:39:13.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368374 '!=' null 2021-05-12T09:39:13.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:13.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:13.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:13.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:13.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:13.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:13.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:13.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:13.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:13.924 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:13.924 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:14.305 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:14.305 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:14.306 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:14.306 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:14.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:14.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:14.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:14.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:14.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:14.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:14.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:14.800 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:14.800 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:15.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863091 2021-05-12T09:39:15.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863091 '!=' null 2021-05-12T09:39:15.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:15.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:15.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:15.401 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:15.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:15.403 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:15.403 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:15.403 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:15.404 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:15.404 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:15.404 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:15.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:15.776 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:15.776 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:15.776 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:16.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:16.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:16.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:16.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:16.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:16.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:16.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:16.277 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:16.277 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:16.779 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242012 2021-05-12T09:39:16.779 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242012 '!=' null 2021-05-12T09:39:16.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:16.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:16.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:16.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:16.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:16.882 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:16.883 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:16.883 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:16.884 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:16.884 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:16.884 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:17.251 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:17.253 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:17.253 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:17.253 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:17.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:17.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:17.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:17.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:17.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:17.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:17.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:17.773 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:17.773 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:18.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242014 2021-05-12T09:39:18.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242014 '!=' null 2021-05-12T09:39:18.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:18.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:18.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:18.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:18.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:18.372 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:18.373 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:18.373 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:18.373 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:18.373 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:18.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:18.738 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:18.738 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:18.739 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:18.739 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:19.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:19.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:19.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:19.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:19.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:19.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:19.256 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:19.257 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:19.257 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:19.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242014 2021-05-12T09:39:19.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242014 '!=' null 2021-05-12T09:39:19.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:19.753 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:19.753 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:19.853 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:19.853 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:19.854 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:19.855 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:19.855 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:19.855 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:19.855 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:19.856 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:20.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:20.227 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:20.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:20.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:20.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:20.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:20.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:20.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:20.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:20.734 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:20.734 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:20.734 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:20.734 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:21.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863090 2021-05-12T09:39:21.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863090 '!=' null 2021-05-12T09:39:21.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:21.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:21.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:21.333 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:21.334 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:21.336 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:21.336 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:21.336 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:21.336 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:21.336 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:21.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:21.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:21.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:21.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:21.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:22.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:22.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:22.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:22.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:22.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:22.221 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:22.221 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:22.221 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:22.221 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:22.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115689 2021-05-12T09:39:22.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115689 '!=' null 2021-05-12T09:39:22.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:22.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:22.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:22.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:22.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:22.825 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:22.825 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:22.825 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:22.825 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:22.826 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:22.826 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:23.211 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:23.212 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:23.212 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:23.212 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:23.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:23.711 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:23.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:23.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:23.712 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:23.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:23.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:23.716 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:23.716 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:24.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863027 2021-05-12T09:39:24.300 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863027 '!=' null 2021-05-12T09:39:24.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:24.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:24.301 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:24.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:24.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:24.400 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:24.401 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:24.401 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:24.401 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:24.401 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:24.401 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:24.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:24.771 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:24.771 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:24.772 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:25.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:25.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:25.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:25.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:25.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:25.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:25.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:25.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:25.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:25.799 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863027 2021-05-12T09:39:25.800 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863027 '!=' null 2021-05-12T09:39:25.800 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:25.800 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:25.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:25.902 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:25.902 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:25.902 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:25.902 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:25.903 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:25.903 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:25.903 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:25.903 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:26.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:26.270 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:26.271 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:26.271 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:26.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:26.773 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:26.773 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:26.773 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:26.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:26.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:26.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:26.775 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:26.775 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:27.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368303 2021-05-12T09:39:27.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368303 '!=' null 2021-05-12T09:39:27.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:27.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:27.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:27.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:27.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:27.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:27.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:27.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:27.381 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:27.381 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:27.381 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:27.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:27.751 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:27.751 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:27.751 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:28.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:28.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:28.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:28.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:28.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:28.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:28.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:28.265 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:28.265 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:28.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368312 2021-05-12T09:39:28.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368312 '!=' null 2021-05-12T09:39:28.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:28.773 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:28.773 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:28.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:28.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:28.873 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:28.873 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:28.873 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:28.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:28.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:28.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:29.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:29.240 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:29.240 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:29.241 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:29.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:29.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:29.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:29.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:29.753 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:29.753 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:29.756 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:29.757 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:29.757 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:30.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989329 2021-05-12T09:39:30.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989329 '!=' null 2021-05-12T09:39:30.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:30.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:30.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:30.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:30.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:30.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:30.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:30.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:30.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:30.375 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:30.375 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:30.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:30.746 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:30.747 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:30.747 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:31.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:31.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:31.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:31.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:31.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:31.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:31.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:31.264 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:31.264 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:31.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989329 2021-05-12T09:39:31.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989329 '!=' null 2021-05-12T09:39:31.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:31.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:31.766 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:31.865 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:31.866 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:31.867 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:31.868 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:31.868 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:31.869 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:31.869 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:31.869 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:32.239 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:32.241 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:32.242 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:32.242 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:32.776 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:32.776 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:32.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:32.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:32.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:32.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:32.778 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:32.778 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:32.778 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:33.352 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368284 2021-05-12T09:39:33.353 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368284 '!=' null 2021-05-12T09:39:33.353 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:33.353 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:33.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:33.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:33.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:33.454 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:33.454 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:33.454 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:33.454 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:33.455 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:33.455 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:33.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:33.831 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:33.832 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:33.832 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:34.331 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:34.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:34.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:34.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:34.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:34.333 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:34.333 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:34.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:34.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:34.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989317 2021-05-12T09:39:34.859 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989317 '!=' null 2021-05-12T09:39:34.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:34.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:34.860 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:34.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:34.961 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:34.961 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:34.961 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:34.961 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:34.961 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:34.962 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:34.962 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:35.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:35.329 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:35.330 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:35.331 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:35.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:35.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:35.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:35.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:35.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:35.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:35.832 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:35.833 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:35.833 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:36.338 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241977 2021-05-12T09:39:36.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241977 '!=' null 2021-05-12T09:39:36.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:36.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:36.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:36.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:36.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:36.441 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:36.441 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:36.441 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:36.441 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:36.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:36.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:36.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:36.810 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:36.811 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:36.811 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:37.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:37.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:37.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:37.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:37.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:37.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:37.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:37.316 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:37.317 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:37.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241977 2021-05-12T09:39:37.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241977 '!=' null 2021-05-12T09:39:37.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:37.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:37.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:37.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:37.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:37.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:37.924 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:37.924 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:37.924 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:37.924 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:37.925 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:38.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:38.328 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:38.329 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:38.329 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:38.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:38.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:38.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:38.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:38.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:38.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:38.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:38.836 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:38.836 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:39.333 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242030 2021-05-12T09:39:39.333 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242030 '!=' null 2021-05-12T09:39:39.333 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:39.334 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:39.334 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:39.434 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:39.434 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:39.435 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:39.435 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:39.436 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:39.436 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:39.436 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:39.436 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:39.808 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:39.810 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:39.810 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:39.811 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:40.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:40.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:40.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:40.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:40.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:40.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:40.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:40.315 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:40.315 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:40.806 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989415 2021-05-12T09:39:40.807 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989415 '!=' null 2021-05-12T09:39:40.807 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:40.807 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:40.808 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:40.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:40.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:40.908 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:40.909 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:40.909 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:40.909 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:40.909 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:40.910 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:41.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:41.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:41.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:41.277 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:41.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:41.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:41.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:41.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:41.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:41.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:41.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:41.785 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:41.785 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:42.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368348 2021-05-12T09:39:42.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368348 '!=' null 2021-05-12T09:39:42.305 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:42.305 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:42.305 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:42.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:42.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:42.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:42.407 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:42.407 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:42.407 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:42.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:42.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:42.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:42.775 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:42.775 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:42.776 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:43.287 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:43.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:43.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:43.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:43.288 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:43.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:43.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:43.289 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:43.290 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:43.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368348 2021-05-12T09:39:43.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368348 '!=' null 2021-05-12T09:39:43.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:43.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:43.789 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:43.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:43.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:43.890 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:43.890 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:43.891 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:43.891 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:43.891 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:43.891 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:44.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:44.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:44.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:44.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:44.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:44.833 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:44.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:44.835 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:45.334 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989397 2021-05-12T09:39:45.334 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989397 '!=' null 2021-05-12T09:39:45.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:45.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:45.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:45.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:45.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:45.436 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:45.436 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:45.436 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:45.437 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:45.437 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:45.437 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:45.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:45.819 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:45.819 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:45.819 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:46.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:46.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:46.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:46.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:46.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:46.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:46.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:46.315 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:46.316 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:46.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368323 2021-05-12T09:39:46.818 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368323 '!=' null 2021-05-12T09:39:46.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:46.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:46.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:46.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:46.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:46.919 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:46.919 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:46.920 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:46.925 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:46.925 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:46.925 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:47.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:47.296 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:47.297 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:47.297 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:47.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:47.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:47.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:47.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:47.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:47.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:47.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:47.883 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:47.884 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:48.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368341 2021-05-12T09:39:48.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368341 '!=' null 2021-05-12T09:39:48.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:48.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:48.382 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:48.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:48.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:48.481 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:48.481 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:48.481 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:48.482 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:48.482 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:48.482 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:48.847 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:48.848 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:48.849 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:48.849 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:49.364 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:49.364 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:49.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:49.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:49.365 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:49.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:49.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:49.367 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:49.367 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:49.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862972 2021-05-12T09:39:49.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862972 '!=' null 2021-05-12T09:39:49.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:49.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:49.868 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:49.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:49.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:49.969 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:49.969 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:49.969 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:49.970 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:49.970 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:49.970 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:50.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:50.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:50.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:50.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:50.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:50.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:50.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:50.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:50.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:50.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:50.838 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:50.839 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:50.839 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:51.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862972 2021-05-12T09:39:51.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862972 '!=' null 2021-05-12T09:39:51.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:51.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:51.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:51.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:51.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:51.445 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:51.445 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:51.446 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:51.446 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:51.447 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:51.447 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:51.813 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:51.814 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:51.814 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:51.814 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:52.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:52.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:52.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:52.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:52.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:52.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:52.316 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:52.317 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:52.317 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:52.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115636 2021-05-12T09:39:52.814 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115636 '!=' null 2021-05-12T09:39:52.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:52.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:52.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:52.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:52.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:52.916 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:52.917 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:52.917 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:52.917 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:52.918 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:52.918 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:53.289 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:53.290 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:53.291 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:53.291 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:53.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:53.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:53.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:53.792 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:53.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:53.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:53.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:53.794 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:53.794 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:54.296 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736670 2021-05-12T09:39:54.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736670 '!=' null 2021-05-12T09:39:54.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:54.297 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:54.298 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:54.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:54.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:54.398 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:54.398 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:54.398 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:54.398 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:54.399 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:54.399 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:54.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:54.772 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:54.772 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:54.772 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:55.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:55.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:55.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:55.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:55.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:55.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:55.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:55.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:55.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:55.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736670 2021-05-12T09:39:55.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736670 '!=' null 2021-05-12T09:39:55.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:55.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:55.822 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:55.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:55.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:55.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:55.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:55.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:55.924 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:55.924 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:55.924 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:56.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:56.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:56.338 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:56.338 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:56.840 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:56.840 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:56.840 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:56.841 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:56.841 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:56.841 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:56.841 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:56.842 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:56.842 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:57.343 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115675 2021-05-12T09:39:57.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115675 '!=' null 2021-05-12T09:39:57.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:57.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:57.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:57.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:57.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:57.445 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:57.445 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:57.446 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:57.446 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:57.446 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:57.446 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:57.815 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:57.816 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:57.817 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:57.817 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:58.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:58.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:58.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:58.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:58.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:58.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:58.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:58.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:58.337 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:39:58.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115739 2021-05-12T09:39:58.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115739 '!=' null 2021-05-12T09:39:58.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:39:58.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:39:58.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:39:58.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:39:58.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:39:58.931 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:39:58.931 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:39:58.931 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:39:58.931 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:39:58.931 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:39:58.932 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:39:59.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:39:59.325 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:39:59.325 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:39:59.326 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:39:59.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:39:59.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:39:59.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:39:59.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:39:59.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:39:59.829 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:39:59.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:39:59.830 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:39:59.831 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:00.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863083 2021-05-12T09:40:00.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863083 '!=' null 2021-05-12T09:40:00.330 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:00.331 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:00.331 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:00.430 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:00.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:00.432 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:00.432 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:00.432 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:00.432 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:00.433 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:00.434 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:00.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:00.797 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:00.798 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:00.798 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:01.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:01.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:01.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:01.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:01.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:01.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:01.308 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:01.309 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:01.309 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:01.843 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863083 2021-05-12T09:40:01.843 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863083 '!=' null 2021-05-12T09:40:01.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:01.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:01.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:01.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:01.944 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:01.945 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:01.945 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:01.945 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:01.946 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:01.946 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:01.946 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:02.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:02.316 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:02.316 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:02.316 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:02.835 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:02.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:02.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:02.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:02.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:02.836 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:02.837 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:02.837 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:02.838 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:03.339 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368376 2021-05-12T09:40:03.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368376 '!=' null 2021-05-12T09:40:03.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:03.340 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:03.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:03.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:03.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:03.441 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:03.441 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:03.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:03.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:03.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:03.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:03.817 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:03.818 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:03.818 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:03.818 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:04.321 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:04.322 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:04.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:04.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:04.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:04.324 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:04.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:04.325 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:04.325 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:04.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989413 2021-05-12T09:40:04.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989413 '!=' null 2021-05-12T09:40:04.845 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:04.845 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:04.845 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:04.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:04.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:04.946 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:04.946 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:04.950 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:04.950 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:04.950 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:04.950 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:05.321 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:05.322 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:05.323 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:05.323 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:05.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:05.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:05.830 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:05.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:05.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:05.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:05.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:05.832 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:05.832 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:06.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368387 2021-05-12T09:40:06.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368387 '!=' null 2021-05-12T09:40:06.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:06.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:06.329 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:06.429 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:06.429 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:06.430 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:06.431 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:06.431 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:06.431 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:06.431 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:06.432 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:06.819 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:06.820 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:06.820 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:06.820 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:07.313 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:07.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:07.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:07.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:07.314 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:07.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:07.315 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:07.316 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:07.317 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:07.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368387 2021-05-12T09:40:07.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368387 '!=' null 2021-05-12T09:40:07.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:07.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:07.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:07.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:07.924 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:07.925 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:07.925 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:07.925 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:07.925 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:07.926 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:07.926 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:08.295 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:08.296 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:08.297 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:08.297 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:08.801 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:08.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:08.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:08.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:08.802 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:08.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:08.803 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:08.803 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:08.803 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:09.303 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368364 2021-05-12T09:40:09.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368364 '!=' null 2021-05-12T09:40:09.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:09.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:09.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:09.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:09.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:09.405 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:09.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:09.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:09.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:09.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:09.407 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:09.793 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:09.794 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:09.794 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:09.795 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:10.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:10.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:10.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:10.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:10.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:10.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:10.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:10.437 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:10.438 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:10.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115709 2021-05-12T09:40:10.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115709 '!=' null 2021-05-12T09:40:10.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:10.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:10.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:11.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:11.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:11.048 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:11.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:11.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:11.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:11.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:11.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:11.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:11.423 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:11.424 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:11.424 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:11.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:11.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:11.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:11.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:11.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:11.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:11.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:11.928 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:11.929 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:12.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368351 2021-05-12T09:40:12.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368351 '!=' null 2021-05-12T09:40:12.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:12.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:12.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:12.575 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:12.576 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:12.577 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:12.577 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:12.578 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:12.578 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:12.578 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:12.578 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:12.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:12.968 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:12.969 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:12.969 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:13.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:13.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:13.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:13.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:13.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:13.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:13.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:13.482 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:13.482 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:14.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989336 2021-05-12T09:40:14.008 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989336 '!=' null 2021-05-12T09:40:14.008 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:14.008 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:14.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:14.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:14.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:14.110 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:14.110 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:14.110 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:14.110 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:14.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:14.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:14.477 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:14.478 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:14.478 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:14.479 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:14.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:14.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:14.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:14.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:14.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:14.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:14.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:14.987 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:14.987 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:15.486 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989336 2021-05-12T09:40:15.486 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989336 '!=' null 2021-05-12T09:40:15.486 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:15.487 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:15.487 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:15.586 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:15.587 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:15.588 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:15.588 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:15.588 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:15.588 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:15.589 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:15.589 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:15.957 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:15.958 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:15.958 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:15.958 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:16.468 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:16.468 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:16.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:16.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:16.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:16.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:16.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:16.470 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:16.470 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:16.977 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368286 2021-05-12T09:40:16.977 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368286 '!=' null 2021-05-12T09:40:16.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:16.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:16.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:17.078 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:17.079 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:17.080 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:17.080 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:17.080 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:17.080 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:17.081 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:17.081 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:17.451 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:17.452 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:17.452 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:17.452 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:17.994 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:17.994 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:17.994 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:17.994 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:17.996 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:17.996 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:17.996 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:17.997 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:17.997 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:18.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368281 2021-05-12T09:40:18.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368281 '!=' null 2021-05-12T09:40:18.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:18.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:18.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:18.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:18.597 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:18.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:18.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:18.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:18.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:18.599 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:18.599 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:18.964 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:18.965 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:18.965 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:18.966 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:19.467 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:19.467 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:19.467 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:19.467 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:19.468 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:19.468 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:19.468 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:19.469 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:19.469 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:20.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989295 2021-05-12T09:40:20.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989295 '!=' null 2021-05-12T09:40:20.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:20.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:20.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:20.102 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:20.102 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:20.104 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:20.104 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:20.104 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:20.105 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:20.105 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:20.105 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:20.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:20.476 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:20.476 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:20.476 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:20.977 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:20.977 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:20.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:20.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:20.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:20.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:20.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:20.979 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:20.980 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:21.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989295 2021-05-12T09:40:21.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989295 '!=' null 2021-05-12T09:40:21.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:21.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:21.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:21.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:21.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:21.597 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:21.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:21.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:21.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:21.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:21.599 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:21.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:21.971 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:21.972 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:21.972 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:22.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:22.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:22.472 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:22.472 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:22.472 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:22.472 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:22.472 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:22.473 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:22.473 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:22.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241972 2021-05-12T09:40:22.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241972 '!=' null 2021-05-12T09:40:22.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:22.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:22.970 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:23.070 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:23.071 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:23.072 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:23.072 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:23.072 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:23.073 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:23.073 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:23.073 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:23.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:23.443 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:23.443 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:23.443 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:23.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:23.941 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:23.941 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:23.941 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:23.941 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:23.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:23.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:23.942 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:23.943 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:24.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862988 2021-05-12T09:40:24.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862988 '!=' null 2021-05-12T09:40:24.439 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:24.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:24.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:24.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:24.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:24.542 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:24.542 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:24.543 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:24.544 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:24.544 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:24.544 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:24.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:25.026 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:25.027 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:25.027 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:25.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:25.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:25.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:25.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:25.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:25.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:25.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:25.418 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:25.419 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:25.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242033 2021-05-12T09:40:25.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242033 '!=' null 2021-05-12T09:40:25.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:25.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:25.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:26.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:26.029 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:26.030 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:26.030 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:26.030 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:26.031 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:26.031 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:26.031 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:26.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:26.401 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:26.401 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:26.401 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:26.900 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:26.900 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:26.901 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:26.901 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:26.901 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:26.901 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:26.902 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:26.902 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:26.903 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:27.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242033 2021-05-12T09:40:27.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242033 '!=' null 2021-05-12T09:40:27.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:27.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:27.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:27.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:27.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:27.509 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:27.509 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:27.509 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:27.510 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:27.510 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:27.510 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:27.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:27.879 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:27.880 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:27.880 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:28.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:28.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:28.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:28.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:28.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:28.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:28.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:28.399 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:28.400 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:28.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242089 2021-05-12T09:40:28.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242089 '!=' null 2021-05-12T09:40:28.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:28.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:28.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:29.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:29.069 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:29.070 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:29.070 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:29.070 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:29.071 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:29.071 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:29.071 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:29.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:29.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:29.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:29.443 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:29.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:29.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:29.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:29.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:29.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:29.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:29.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:29.948 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:29.948 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:30.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863115 2021-05-12T09:40:30.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863115 '!=' null 2021-05-12T09:40:30.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:30.458 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:30.458 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:30.557 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:30.558 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:30.559 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:30.559 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:30.559 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:30.560 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:30.560 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:30.560 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:30.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:30.930 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:30.930 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:30.931 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:31.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:31.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:31.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:31.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:31.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:31.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:31.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:31.443 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:31.443 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:31.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863115 2021-05-12T09:40:31.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863115 '!=' null 2021-05-12T09:40:31.945 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:31.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:31.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:32.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:32.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:32.047 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:32.047 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:32.047 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:32.047 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:32.048 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:32.048 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:32.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:32.418 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:32.418 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:32.418 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:32.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:32.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:32.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:32.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:32.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:32.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:32.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:32.928 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:32.929 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:33.425 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242077 2021-05-12T09:40:33.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242077 '!=' null 2021-05-12T09:40:33.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:33.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:33.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:33.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:33.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:33.527 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:33.528 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:33.528 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:33.529 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:33.529 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:33.529 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:33.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:33.905 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:33.905 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:33.905 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:34.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:34.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:34.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:34.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:34.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:34.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:34.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:34.423 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:34.423 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:34.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863074 2021-05-12T09:40:34.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863074 '!=' null 2021-05-12T09:40:34.923 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:34.924 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:34.924 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:35.023 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:35.024 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:35.025 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:35.025 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:35.025 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:35.026 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:35.026 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:35.026 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:35.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:35.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:35.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:35.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:35.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:35.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:35.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:35.930 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:35.930 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:35.930 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:35.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:35.931 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:35.932 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:36.449 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242005 2021-05-12T09:40:36.449 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242005 '!=' null 2021-05-12T09:40:36.449 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:36.449 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:36.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:36.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:36.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:36.551 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:36.551 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:36.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:36.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:36.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:36.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:36.926 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:36.927 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:36.927 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:36.927 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:37.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:37.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:37.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:37.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:37.437 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:37.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:37.438 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:37.438 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:37.439 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:37.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242009 2021-05-12T09:40:37.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242009 '!=' null 2021-05-12T09:40:37.942 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:37.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:37.943 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:38.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:38.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:38.044 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:38.044 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:38.045 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:38.045 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:38.045 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:38.045 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:38.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:38.419 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:38.419 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:38.419 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:38.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:38.920 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:38.920 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:38.920 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:38.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:38.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:38.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:38.922 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:38.922 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:39.456 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242009 2021-05-12T09:40:39.456 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242009 '!=' null 2021-05-12T09:40:39.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:39.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:39.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:39.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:39.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:39.557 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:39.558 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:39.558 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:39.558 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:39.558 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:39.559 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:39.937 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:39.938 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:39.938 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:39.939 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:40.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:40.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:40.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:40.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:40.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:40.443 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:40.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:40.444 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:40.445 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:40.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989346 2021-05-12T09:40:40.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989346 '!=' null 2021-05-12T09:40:40.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:40.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:40.954 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:41.049 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:41.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:41.051 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:41.051 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:41.051 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:41.052 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:41.052 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:41.052 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:41.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:41.438 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:41.438 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:41.439 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:41.933 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:41.933 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:41.934 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:41.934 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:41.934 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:41.935 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:41.935 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:41.936 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:41.936 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:42.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368346 2021-05-12T09:40:42.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368346 '!=' null 2021-05-12T09:40:42.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:42.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:42.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:42.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:42.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:42.555 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:42.556 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:42.556 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:42.556 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:42.556 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:42.557 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:42.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:42.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:42.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:42.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:43.429 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:43.430 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:43.430 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:43.430 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:43.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:43.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:43.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:43.432 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:43.432 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:43.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989362 2021-05-12T09:40:43.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989362 '!=' null 2021-05-12T09:40:43.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:43.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:43.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:44.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:44.058 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:44.059 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:44.060 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:44.060 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:44.060 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:44.060 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:44.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:44.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:44.428 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:44.428 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:44.428 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:44.933 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:44.934 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:44.934 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:44.934 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:44.935 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:44.935 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:44.935 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:44.936 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:44.936 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:45.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989362 2021-05-12T09:40:45.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989362 '!=' null 2021-05-12T09:40:45.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:45.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:45.436 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:45.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:45.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:45.537 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:45.538 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:45.538 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:45.538 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:45.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:45.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:45.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:45.911 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:45.911 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:45.911 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:46.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:46.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:46.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:46.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:46.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:46.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:46.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:46.421 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:46.421 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:46.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368332 2021-05-12T09:40:46.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368332 '!=' null 2021-05-12T09:40:46.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:46.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:46.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:47.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:47.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:47.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:47.016 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:47.016 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:47.017 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:47.017 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:47.017 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:47.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:47.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:47.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:47.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:47.887 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:47.887 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:47.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:47.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:47.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:47.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:47.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:47.889 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:47.889 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:48.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368362 2021-05-12T09:40:48.392 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368362 '!=' null 2021-05-12T09:40:48.392 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:48.394 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:48.394 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:48.489 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:48.489 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:48.493 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:48.493 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:48.493 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:48.494 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:48.494 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:48.494 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:48.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:48.861 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:48.862 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:48.862 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:49.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:49.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:49.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:49.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:49.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:49.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:49.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:49.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:49.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:49.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989389 2021-05-12T09:40:49.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989389 '!=' null 2021-05-12T09:40:49.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:49.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:49.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:49.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:49.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:49.986 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:49.986 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:49.986 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:49.986 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:49.986 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:49.987 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:50.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:50.354 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:50.355 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:50.355 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:50.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:50.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:50.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:50.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:50.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:50.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:50.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:50.857 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:50.857 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:51.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989389 2021-05-12T09:40:51.355 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989389 '!=' null 2021-05-12T09:40:51.355 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:51.355 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:51.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:51.455 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:51.455 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:51.456 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:51.457 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:51.457 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:51.457 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:51.457 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:51.458 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:51.831 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:51.831 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:51.832 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:51.832 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:52.334 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:52.334 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:52.335 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:52.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:52.336 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:52.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:52.337 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:52.338 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:52.339 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:52.847 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242050 2021-05-12T09:40:52.847 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242050 '!=' null 2021-05-12T09:40:52.848 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:52.848 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:52.848 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:52.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:52.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:52.949 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:52.949 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:52.950 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:52.950 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:52.950 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:52.950 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:53.322 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:53.322 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:53.322 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:53.323 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:53.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:53.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:53.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:53.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:53.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:53.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:53.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:53.857 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:53.857 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:54.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989370 2021-05-12T09:40:54.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989370 '!=' null 2021-05-12T09:40:54.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:54.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:54.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:54.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:54.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:54.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:54.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:54.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:54.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:54.501 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:54.501 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:54.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:54.871 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:54.872 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:54.872 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:55.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:55.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:55.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:55.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:55.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:55.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:55.373 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:55.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:55.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:55.877 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368336 2021-05-12T09:40:55.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368336 '!=' null 2021-05-12T09:40:55.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:55.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:55.879 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:55.977 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:55.978 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:55.979 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:55.979 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:55.979 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:55.980 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:55.980 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:55.980 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:56.349 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:56.349 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:56.350 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:56.350 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:56.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:56.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:56.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:56.861 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:56.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:56.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:56.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:56.862 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:56.863 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:57.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368336 2021-05-12T09:40:57.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368336 '!=' null 2021-05-12T09:40:57.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:57.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:57.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:57.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:57.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:57.508 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:57.508 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:57.509 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:57.509 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:57.509 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:57.509 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:57.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:57.879 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:57.880 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:57.880 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:58.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:58.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:58.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:58.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:58.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:58.401 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:58.401 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:58.401 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:58.402 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:40:58.902 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368381 2021-05-12T09:40:58.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368381 '!=' null 2021-05-12T09:40:58.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:40:58.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:40:58.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:40:59.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:40:59.003 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:40:59.004 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:40:59.005 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:40:59.005 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:40:59.005 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:40:59.006 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:40:59.006 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:40:59.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:40:59.385 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:40:59.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:40:59.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:40:59.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:40:59.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:40:59.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:40:59.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:40:59.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:40:59.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:40:59.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:40:59.886 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:40:59.887 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:00.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989422 2021-05-12T09:41:00.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989422 '!=' null 2021-05-12T09:41:00.410 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:00.410 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:00.410 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:00.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:00.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:00.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:00.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:00.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:00.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:00.512 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:00.512 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:00.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:00.891 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:00.891 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:00.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:01.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:01.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:01.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:01.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:01.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:01.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:01.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:01.407 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:01.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:01.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368394 2021-05-12T09:41:01.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368394 '!=' null 2021-05-12T09:41:01.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:01.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:01.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:02.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:02.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:02.014 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:02.014 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:02.014 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:02.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:02.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:02.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:02.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:02.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:02.385 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:02.385 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:02.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:02.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:02.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:02.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:02.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:02.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:02.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:02.911 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:02.912 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:03.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368394 2021-05-12T09:41:03.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368394 '!=' null 2021-05-12T09:41:03.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:03.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:03.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:03.519 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:03.520 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:03.521 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:03.521 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:03.521 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:03.521 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:03.522 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:03.522 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:03.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:03.896 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:03.896 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:03.897 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:04.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:04.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:04.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:04.410 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:04.410 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:04.410 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:04.411 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:04.411 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:04.411 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:04.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863123 2021-05-12T09:41:04.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863123 '!=' null 2021-05-12T09:41:04.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:04.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:04.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:05.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:05.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:05.008 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:05.008 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:05.009 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:05.009 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:05.009 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:05.009 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:05.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:05.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:05.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:05.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:05.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:05.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:05.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:05.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:05.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:05.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:05.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:05.891 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:05.891 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:06.416 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115759 2021-05-12T09:41:06.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115759 '!=' null 2021-05-12T09:41:06.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:06.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:06.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:06.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:06.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:06.518 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:06.519 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:06.519 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:06.519 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:06.519 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:06.520 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:06.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:06.889 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:06.889 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:06.889 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:07.395 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:07.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:07.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:07.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:07.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:07.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:07.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:07.397 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:07.398 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:07.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115759 2021-05-12T09:41:07.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115759 '!=' null 2021-05-12T09:41:07.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:07.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:07.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:08.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:08.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:08.010 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:08.011 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:08.011 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:08.011 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:08.011 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:08.012 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:08.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:08.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:08.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:08.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:08.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:08.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:08.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:08.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:08.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:08.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:08.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:08.891 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:08.891 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:09.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115786 2021-05-12T09:41:09.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115786 '!=' null 2021-05-12T09:41:09.391 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:09.392 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:09.392 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:09.492 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:09.493 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:09.494 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:09.494 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:09.494 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:09.495 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:09.495 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:09.495 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:09.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:09.871 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:09.871 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:09.871 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:10.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:10.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:10.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:10.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:10.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:10.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:10.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:10.385 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:10.385 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:10.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736751 2021-05-12T09:41:10.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736751 '!=' null 2021-05-12T09:41:10.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:10.889 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:10.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:10.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:10.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:10.990 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:10.991 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:10.991 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:10.991 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:10.991 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:10.993 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:11.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:11.366 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:11.367 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:11.367 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:11.869 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:11.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:11.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:11.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:11.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:11.870 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:11.871 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:11.871 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:11.872 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:12.398 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115711 2021-05-12T09:41:12.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115711 '!=' null 2021-05-12T09:41:12.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:12.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:12.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:12.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:12.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:12.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:12.501 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:12.501 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:12.501 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:12.501 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:12.502 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:12.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:12.873 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:12.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:12.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:13.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:13.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:13.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:13.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:13.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:13.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:13.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:13.377 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:13.378 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:13.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862950 2021-05-12T09:41:13.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862950 '!=' null 2021-05-12T09:41:13.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:13.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:13.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:13.975 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:13.975 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:13.977 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:13.978 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:13.978 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:13.978 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:13.978 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:13.978 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:14.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:14.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:14.347 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:14.347 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:14.848 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:14.848 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:14.849 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:14.849 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:14.849 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:14.849 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:14.849 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:14.850 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:14.850 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:15.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862950 2021-05-12T09:41:15.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862950 '!=' null 2021-05-12T09:41:15.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:15.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:15.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:15.472 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:15.473 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:15.474 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:15.474 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:15.474 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:15.474 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:15.475 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:15.475 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:15.849 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:15.857 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:15.858 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:15.858 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:16.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:16.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:16.431 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:16.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:16.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:16.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:16.432 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:16.433 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:16.433 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:16.932 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368295 2021-05-12T09:41:16.933 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368295 '!=' null 2021-05-12T09:41:16.934 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:16.934 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:16.935 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:17.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:17.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:17.033 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:17.033 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:17.033 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:17.034 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:17.034 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:17.034 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:17.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:17.410 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:17.410 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:17.410 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:17.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:17.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:17.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:17.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:17.908 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:17.908 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:17.909 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:17.909 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:17.910 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:18.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368286 2021-05-12T09:41:18.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368286 '!=' null 2021-05-12T09:41:18.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:18.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:18.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:18.504 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:18.505 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:18.506 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:18.506 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:18.507 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:18.507 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:18.507 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:18.507 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:18.878 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:18.879 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:18.880 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:18.880 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:19.403 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:19.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:19.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:19.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:19.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:19.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:19.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:19.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:19.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:19.924 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368286 2021-05-12T09:41:19.925 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368286 '!=' null 2021-05-12T09:41:19.925 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:19.925 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:19.925 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:20.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:20.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:20.026 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:20.026 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:20.027 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:20.027 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:20.027 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:20.027 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:20.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:20.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:20.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:20.407 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:20.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:20.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:20.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:20.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:20.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:20.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:20.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:20.918 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:20.918 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:21.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989262 2021-05-12T09:41:21.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989262 '!=' null 2021-05-12T09:41:21.422 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:21.423 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:21.424 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:21.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:21.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:21.524 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:21.524 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:21.524 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:21.524 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:21.525 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:21.525 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:21.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:21.897 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:21.897 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:21.897 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:22.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:22.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:22.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:22.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:22.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:22.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:22.401 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:22.402 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:22.402 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:22.899 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368326 2021-05-12T09:41:22.899 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368326 '!=' null 2021-05-12T09:41:22.899 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:22.900 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:22.900 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:22.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:23.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:23.001 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:23.001 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:23.001 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:23.001 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:23.002 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:23.002 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:23.385 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:23.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:23.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:23.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:23.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:23.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:23.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:23.948 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:23.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:23.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:23.950 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:23.950 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:23.950 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:24.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115656 2021-05-12T09:41:24.444 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115656 '!=' null 2021-05-12T09:41:24.445 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:24.445 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:24.445 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:24.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:24.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:24.546 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:24.547 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:24.547 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:24.547 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:24.548 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:24.548 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:24.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:24.929 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:24.929 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:24.930 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:25.426 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:25.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:25.427 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:25.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:25.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:25.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:25.428 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:25.429 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:25.429 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:25.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368370 2021-05-12T09:41:25.932 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368370 '!=' null 2021-05-12T09:41:25.932 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:25.932 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:25.933 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:26.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:26.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:26.034 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:26.035 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:26.035 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:26.035 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:26.035 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:26.036 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:26.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:26.407 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:26.407 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:26.407 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:26.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:26.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:26.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:26.910 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:26.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:26.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:26.911 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:26.911 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:26.912 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:27.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368370 2021-05-12T09:41:27.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368370 '!=' null 2021-05-12T09:41:27.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:27.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:27.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:27.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:27.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:27.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:27.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:27.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:27.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:27.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:27.512 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:27.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:27.882 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:27.882 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:27.882 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:28.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:28.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:28.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:28.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:28.406 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:28.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:28.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:28.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:28.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:28.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368362 2021-05-12T09:41:28.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368362 '!=' null 2021-05-12T09:41:28.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:28.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:28.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:29.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:29.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:29.007 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:29.007 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:29.007 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:29.008 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:29.008 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:29.008 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:29.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:29.385 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:29.385 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:29.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:29.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:29.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:29.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:29.892 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:29.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:29.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:29.893 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:29.894 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:29.894 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:30.401 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989426 2021-05-12T09:41:30.401 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989426 '!=' null 2021-05-12T09:41:30.401 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:30.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:30.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:30.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:30.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:30.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:30.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:30.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:30.504 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:30.504 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:30.504 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:30.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:30.875 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:30.876 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:30.876 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:31.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:31.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:31.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:31.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:31.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:31.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:31.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:31.381 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:31.382 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:31.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368376 2021-05-12T09:41:31.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368376 '!=' null 2021-05-12T09:41:31.883 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:31.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:31.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:31.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:31.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:31.984 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:31.985 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:31.985 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:31.985 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:31.985 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:31.986 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:32.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:32.359 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:32.359 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:32.360 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:32.866 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:32.866 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:32.866 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:32.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:32.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:32.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:32.867 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:32.868 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:32.868 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:33.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368376 2021-05-12T09:41:33.382 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368376 '!=' null 2021-05-12T09:41:33.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:33.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:33.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:33.482 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:33.482 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:33.484 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:33.485 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:33.485 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:33.486 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:33.486 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:33.487 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:33.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:33.905 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:33.906 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:33.906 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:34.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:34.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:34.404 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:34.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:34.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:34.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:34.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:34.405 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:34.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:34.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989414 2021-05-12T09:41:34.912 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989414 '!=' null 2021-05-12T09:41:34.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:34.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:34.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:35.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:35.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:35.014 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:35.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:35.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:35.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:35.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:35.016 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:35.382 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:35.383 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:35.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:35.384 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:35.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:35.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:35.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:35.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:35.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:35.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:35.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:35.886 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:35.886 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:36.414 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242066 2021-05-12T09:41:36.414 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242066 '!=' null 2021-05-12T09:41:36.414 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:36.415 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:36.415 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:36.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:36.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:36.516 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:36.516 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:36.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:36.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:36.518 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:36.518 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:36.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:36.889 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:36.889 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:36.890 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:37.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:37.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:37.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:37.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:37.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:37.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:37.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:37.410 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:37.410 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:37.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242076 2021-05-12T09:41:37.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242076 '!=' null 2021-05-12T09:41:37.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:37.918 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:37.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:38.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:38.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:38.020 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:38.020 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:38.021 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:38.021 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:38.021 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:38.021 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:38.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:38.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:38.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:38.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:38.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:38.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:38.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:38.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:38.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:38.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:38.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:38.916 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:38.916 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:39.414 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242076 2021-05-12T09:41:39.414 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242076 '!=' null 2021-05-12T09:41:39.415 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:39.415 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:39.415 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:39.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:39.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:39.516 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:39.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:39.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:39.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:39.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:39.518 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:39.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:39.906 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:39.907 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:39.907 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:40.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:40.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:40.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:40.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:40.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:40.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:40.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:40.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:40.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:40.905 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863124 2021-05-12T09:41:40.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863124 '!=' null 2021-05-12T09:41:40.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:40.906 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:40.907 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:41.006 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:41.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:41.008 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:41.008 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:41.008 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:41.008 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:41.009 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:41.009 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:41.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:41.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:41.389 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:41.389 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:41.913 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:41.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:41.914 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:41.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:41.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:41.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:41.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:41.916 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:41.916 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:42.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242081 2021-05-12T09:41:42.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242081 '!=' null 2021-05-12T09:41:42.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:42.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:42.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:42.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:42.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:42.562 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:42.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:42.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:42.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:42.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:42.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:42.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:42.931 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:42.932 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:42.932 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:43.434 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:43.434 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:43.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:43.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:43.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:43.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:43.435 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:43.436 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:43.436 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:43.937 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736833 2021-05-12T09:41:43.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736833 '!=' null 2021-05-12T09:41:43.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:43.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:43.938 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:44.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:44.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:44.039 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:44.040 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:44.040 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:44.040 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:44.040 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:44.040 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:44.417 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:44.418 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:44.418 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:44.915 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:44.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:44.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:44.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:44.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:44.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:44.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:44.917 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:44.918 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:45.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736833 2021-05-12T09:41:45.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736833 '!=' null 2021-05-12T09:41:45.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:45.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:45.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:45.519 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:45.519 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:45.520 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:45.521 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:45.521 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:45.522 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:45.522 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:45.522 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:45.894 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:45.895 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:45.896 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:45.896 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:46.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:46.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:46.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:46.408 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:46.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:46.409 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:46.411 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:46.411 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:46.411 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:47.042 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115763 2021-05-12T09:41:47.042 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115763 '!=' null 2021-05-12T09:41:47.042 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:47.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:47.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:47.142 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:47.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:47.144 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:47.144 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:47.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:47.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:47.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:47.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:47.512 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:47.516 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:47.516 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:47.516 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:48.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:48.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:48.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:48.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:48.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:48.049 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:48.049 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:48.050 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:48.050 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:48.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115728 2021-05-12T09:41:48.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115728 '!=' null 2021-05-12T09:41:48.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:48.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:48.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:48.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:48.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:48.663 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:48.664 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:48.664 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:48.664 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:48.664 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:48.664 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:49.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:49.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:49.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:49.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:49.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:49.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:49.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:49.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:49.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:49.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:49.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:49.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:49.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:50.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863078 2021-05-12T09:41:50.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863078 '!=' null 2021-05-12T09:41:50.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:50.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:50.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:50.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:50.153 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:50.154 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:50.155 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:50.155 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:50.155 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:50.156 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:50.156 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:50.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:50.524 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:50.524 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:50.524 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:51.042 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:51.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:51.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:51.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:51.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:51.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:51.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:51.045 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:51.045 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:51.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863078 2021-05-12T09:41:51.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863078 '!=' null 2021-05-12T09:41:51.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:51.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:51.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:51.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:51.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:51.644 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:51.644 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:51.644 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:51.644 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:51.645 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:51.645 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:52.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:52.013 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:52.013 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:52.013 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:52.519 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:52.520 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:52.520 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:52.520 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:52.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:52.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:52.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:52.522 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:52.522 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:53.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242045 2021-05-12T09:41:53.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242045 '!=' null 2021-05-12T09:41:53.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:53.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:53.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:53.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:53.121 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:53.122 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:53.123 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:53.123 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:53.123 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:53.123 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:53.124 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:53.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:53.498 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:53.498 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:53.499 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:54.018 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:54.018 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:54.018 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:54.018 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:54.019 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:54.019 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:54.019 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:54.020 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:54.020 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:54.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863013 2021-05-12T09:41:54.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863013 '!=' null 2021-05-12T09:41:54.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:54.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:54.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:54.633 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:54.634 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:54.635 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:54.635 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:54.635 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:54.635 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:54.636 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:54.636 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:55.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:55.022 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:55.022 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:55.022 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:55.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:55.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:55.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:55.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:55.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:55.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:55.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:55.526 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:55.526 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:56.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368335 2021-05-12T09:41:56.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368335 '!=' null 2021-05-12T09:41:56.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:56.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:56.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:56.140 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:56.141 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:56.142 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:56.142 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:56.142 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:56.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:56.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:56.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:56.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:56.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:56.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:56.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:57.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:57.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:57.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:57.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:57.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:57.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:57.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:57.054 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:57.054 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:57.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368335 2021-05-12T09:41:57.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368335 '!=' null 2021-05-12T09:41:57.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:57.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:57.563 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:57.662 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:57.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:57.664 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:57.664 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:57.664 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:57.665 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:57.665 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:57.665 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:58.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:58.034 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:58.035 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:58.035 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:41:58.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:41:58.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:41:58.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:41:58.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:41:58.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:41:58.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:41:58.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:41:58.537 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:41:58.537 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:41:59.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368313 2021-05-12T09:41:59.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368313 '!=' null 2021-05-12T09:41:59.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:41:59.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:41:59.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:41:59.134 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:41:59.135 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:41:59.136 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:41:59.136 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:41:59.136 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:41:59.136 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:41:59.137 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:41:59.137 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:41:59.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:41:59.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:41:59.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:41:59.512 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:00.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:00.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:00.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:00.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:00.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:00.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:00.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:00.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:00.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:00.565 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115641 2021-05-12T09:42:00.566 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115641 '!=' null 2021-05-12T09:42:00.566 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:00.566 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:00.566 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:00.666 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:00.666 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:00.667 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:00.668 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:00.668 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:00.668 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:00.668 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:00.669 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:01.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:01.037 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:01.037 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:01.038 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:01.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:01.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:01.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:01.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:01.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:01.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:01.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:01.547 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:01.547 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:02.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368315 2021-05-12T09:42:02.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368315 '!=' null 2021-05-12T09:42:02.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:02.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:02.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:02.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:02.146 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:02.147 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:02.147 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:02.147 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:02.147 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:02.148 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:02.148 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:02.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:02.519 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:02.519 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:02.520 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:03.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:03.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:03.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:03.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:03.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:03.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:03.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:03.037 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:03.037 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:03.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368315 2021-05-12T09:42:03.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368315 '!=' null 2021-05-12T09:42:03.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:03.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:03.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:03.641 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:03.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:03.643 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:03.643 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:03.643 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:03.644 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:03.644 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:03.644 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:04.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:04.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:04.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:04.016 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:04.520 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:04.520 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:04.520 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:04.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:04.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:04.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:04.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:04.522 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:04.522 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:05.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989304 2021-05-12T09:42:05.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989304 '!=' null 2021-05-12T09:42:05.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:05.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:05.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:05.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:05.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:05.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:05.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:05.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:05.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:05.146 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:05.146 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:05.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:05.525 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:05.525 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:05.526 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:06.066 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:06.066 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:06.067 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:06.067 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:06.067 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:06.067 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:06.067 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:06.068 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:06.068 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:06.568 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368345 2021-05-12T09:42:06.569 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368345 '!=' null 2021-05-12T09:42:06.569 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:06.569 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:06.569 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:06.668 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:06.669 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:06.670 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:06.670 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:06.670 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:06.670 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:06.671 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:06.671 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:07.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:07.045 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:07.045 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:07.045 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:07.550 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:07.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:07.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:07.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:07.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:07.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:07.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:07.553 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:07.553 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:08.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368317 2021-05-12T09:42:08.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368317 '!=' null 2021-05-12T09:42:08.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:08.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:08.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:08.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:08.163 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:08.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:08.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:08.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:08.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:08.165 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:08.165 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:08.534 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:08.534 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:08.535 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:08.535 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:09.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:09.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:09.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:09.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:09.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:09.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:09.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:09.033 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:09.033 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:09.534 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368317 2021-05-12T09:42:09.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368317 '!=' null 2021-05-12T09:42:09.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:09.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:09.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:09.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:09.635 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:09.637 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:09.637 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:09.637 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:09.638 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:09.638 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:09.638 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:10.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:10.008 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:10.009 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:10.009 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:10.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:10.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:10.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:10.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:10.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:10.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:10.516 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:10.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:10.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:11.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989370 2021-05-12T09:42:11.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989370 '!=' null 2021-05-12T09:42:11.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:11.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:11.018 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:11.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:11.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:11.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:11.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:11.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:11.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:11.120 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:11.120 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:11.485 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:11.486 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:11.486 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:11.487 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:12.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:12.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:12.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:12.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:12.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:12.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:12.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:12.087 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:12.088 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:12.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242019 2021-05-12T09:42:12.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242019 '!=' null 2021-05-12T09:42:12.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:12.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:12.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:12.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:12.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:12.707 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:12.707 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:12.707 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:12.708 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:12.708 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:12.708 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:13.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:13.106 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:13.106 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:13.107 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:13.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:13.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:13.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:13.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:13.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:13.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:13.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:13.608 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:13.608 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:14.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989304 2021-05-12T09:42:14.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989304 '!=' null 2021-05-12T09:42:14.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:14.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:14.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:14.215 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:14.215 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:14.216 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:14.217 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:14.217 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:14.217 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:14.217 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:14.217 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:14.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:14.586 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:14.587 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:14.587 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:15.095 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:15.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:15.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:15.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:15.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:15.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:15.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:15.099 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:15.099 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:15.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989304 2021-05-12T09:42:15.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989304 '!=' null 2021-05-12T09:42:15.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:15.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:15.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:15.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:15.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:15.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:15.707 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:15.707 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:15.707 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:15.707 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:15.708 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:16.079 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:16.081 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:16.081 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:16.081 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:16.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:16.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:16.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:16.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:16.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:16.612 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:16.612 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:16.612 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:16.612 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:17.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368287 2021-05-12T09:42:17.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368287 '!=' null 2021-05-12T09:42:17.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:17.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:17.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:17.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:17.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:17.227 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:17.227 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:17.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:17.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:17.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:17.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:17.639 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:17.639 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:17.640 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:17.640 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:18.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:18.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:18.318 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:18.319 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:18.319 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:18.319 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:18.320 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:18.321 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:18.321 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:18.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368264 2021-05-12T09:42:18.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368264 '!=' null 2021-05-12T09:42:18.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:18.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:18.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:18.920 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:18.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:18.922 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:18.922 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:18.922 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:18.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:18.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:18.923 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:19.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:19.293 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:19.294 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:19.294 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:19.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:19.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:19.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:19.796 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:19.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:19.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:19.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:19.798 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:19.798 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:20.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989232 2021-05-12T09:42:20.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989232 '!=' null 2021-05-12T09:42:20.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:20.305 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:20.305 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:20.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:20.405 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:20.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:20.406 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:20.407 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:20.407 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:20.407 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:20.407 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:20.778 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:20.779 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:20.779 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:20.780 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:21.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:21.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:21.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:21.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:21.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:21.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:21.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:21.281 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:21.281 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:21.787 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989232 2021-05-12T09:42:21.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989232 '!=' null 2021-05-12T09:42:21.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:21.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:21.788 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:21.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:21.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:21.889 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:21.890 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:21.890 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:21.890 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:21.890 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:21.891 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:22.392 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:22.393 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:22.393 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:22.394 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:22.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:22.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:22.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:22.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:22.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:22.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:22.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:22.898 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:22.899 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:23.396 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368239 2021-05-12T09:42:23.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368239 '!=' null 2021-05-12T09:42:23.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:23.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:23.397 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:23.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:23.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:23.499 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:23.499 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:23.499 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:23.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:23.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:23.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:23.936 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:23.938 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:23.938 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:23.938 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:24.448 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:24.448 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:24.448 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:24.449 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:24.449 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:24.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:24.450 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:24.451 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:24.451 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:24.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862835 2021-05-12T09:42:24.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862835 '!=' null 2021-05-12T09:42:24.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:24.950 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:24.950 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:25.049 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:25.050 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:25.051 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:25.051 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:25.052 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:25.052 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:25.052 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:25.052 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:25.421 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:25.421 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:25.422 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:25.422 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:25.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:25.951 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:25.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:25.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:25.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:25.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:25.953 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:25.953 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:25.954 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:26.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115569 2021-05-12T09:42:26.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115569 '!=' null 2021-05-12T09:42:26.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:26.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:26.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:26.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:26.597 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:26.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:26.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:26.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:26.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:26.599 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:26.599 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:26.977 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:26.977 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:26.978 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:26.978 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:27.493 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:27.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:27.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:27.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:27.494 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:27.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:27.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:27.496 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:27.496 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:28.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115600 2021-05-12T09:42:28.055 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115600 '!=' null 2021-05-12T09:42:28.055 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:28.055 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:28.055 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:28.155 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:28.155 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:28.156 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:28.156 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:28.157 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:28.157 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:28.157 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:28.157 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:28.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:28.527 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:28.527 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:28.528 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:29.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:29.026 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:29.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:29.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:29.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:29.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:29.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:29.029 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:29.029 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:29.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115600 2021-05-12T09:42:29.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115600 '!=' null 2021-05-12T09:42:29.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:29.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:29.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:29.633 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:29.633 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:29.634 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:29.634 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:29.635 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:29.635 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:29.635 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:29.635 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:30.019 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:30.020 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:30.021 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:30.021 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:30.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:30.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:30.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:30.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:30.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:30.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:30.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:30.531 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:30.531 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:31.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736612 2021-05-12T09:42:31.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736612 '!=' null 2021-05-12T09:42:31.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:31.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:31.036 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:31.136 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:31.137 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:31.138 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:31.138 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:31.138 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:31.138 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:31.139 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:31.139 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:31.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:31.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:31.504 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:31.504 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:32.015 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:32.015 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:32.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:32.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:32.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:32.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:32.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:32.017 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:32.017 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:32.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115643 2021-05-12T09:42:32.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115643 '!=' null 2021-05-12T09:42:32.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:32.519 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:32.519 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:32.619 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:32.619 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:32.620 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:32.621 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:32.621 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:32.622 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:32.622 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:32.622 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:32.988 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:32.989 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:32.989 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:32.990 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:33.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:33.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:33.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:33.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:33.497 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:33.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:33.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:33.498 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:33.499 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:34.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863017 2021-05-12T09:42:34.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863017 '!=' null 2021-05-12T09:42:34.000 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:34.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:34.002 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:34.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:34.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:34.102 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:34.102 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:34.103 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:34.103 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:34.103 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:34.103 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:34.476 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:34.477 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:34.477 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:34.478 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:34.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:34.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:34.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:34.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:34.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:34.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:34.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:34.984 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:34.984 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:35.520 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863017 2021-05-12T09:42:35.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863017 '!=' null 2021-05-12T09:42:35.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:35.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:35.521 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:35.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:35.621 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:35.622 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:35.622 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:35.623 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:35.623 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:35.623 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:35.623 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:35.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:36.000 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:36.000 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:36.000 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:36.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:36.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:36.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:36.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:36.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:36.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:36.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:36.502 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:36.502 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:37.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368325 2021-05-12T09:42:37.006 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368325 '!=' null 2021-05-12T09:42:37.006 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:37.006 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:37.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:37.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:37.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:37.107 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:37.108 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:37.108 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:37.108 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:37.108 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:37.109 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:37.473 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:37.474 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:37.474 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:37.474 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:37.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:37.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:37.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:37.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:37.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:37.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:37.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:37.970 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:37.970 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:38.482 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368337 2021-05-12T09:42:38.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368337 '!=' null 2021-05-12T09:42:38.483 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:38.484 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:38.484 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:38.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:38.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:38.584 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:38.584 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:38.585 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:38.585 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:38.585 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:38.586 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:38.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:38.959 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:38.959 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:38.959 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:39.559 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:39.559 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:39.559 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:39.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:39.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:39.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:39.560 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:39.561 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:39.561 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:40.067 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989328 2021-05-12T09:42:40.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989328 '!=' null 2021-05-12T09:42:40.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:40.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:40.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:40.168 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:40.168 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:40.169 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:40.169 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:40.170 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:40.171 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:40.171 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:40.171 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:40.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:40.544 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:40.545 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:40.545 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:41.051 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:41.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:41.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:41.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:41.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:41.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:41.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:41.054 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:41.054 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:41.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989328 2021-05-12T09:42:41.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989328 '!=' null 2021-05-12T09:42:41.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:41.552 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:41.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:41.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:41.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:41.654 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:41.654 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:41.654 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:41.654 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:41.655 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:41.655 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:42.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:42.036 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:42.037 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:42.037 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:42.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:42.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:42.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:42.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:42.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:42.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:42.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:42.541 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:42.542 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:43.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368339 2021-05-12T09:42:43.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368339 '!=' null 2021-05-12T09:42:43.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:43.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:43.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:43.141 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:43.141 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:43.142 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:43.142 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:43.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:43.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:43.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:43.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:43.511 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:43.512 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:43.512 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:43.513 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:44.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:44.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:44.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:44.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:44.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:44.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:44.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:44.047 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:44.047 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115694 2021-05-12T09:42:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115694 '!=' null 2021-05-12T09:42:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:44.661 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:44.661 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:44.662 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:44.662 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:44.663 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:44.663 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:45.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:45.031 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:45.031 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:45.031 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:45.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:45.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:45.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:45.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:45.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:45.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:45.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:45.531 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:45.532 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:46.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368353 2021-05-12T09:42:46.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368353 '!=' null 2021-05-12T09:42:46.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:46.042 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:46.042 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:46.141 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:46.141 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:46.142 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:46.142 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:46.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:46.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:46.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:46.143 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:46.513 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:46.514 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:46.515 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:46.515 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:47.023 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:47.024 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:47.024 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:47.024 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:47.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:47.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:47.025 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:47.026 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:47.026 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:47.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368353 2021-05-12T09:42:47.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368353 '!=' null 2021-05-12T09:42:47.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:47.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:47.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:47.622 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:47.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:47.624 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:47.624 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:47.624 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:47.624 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:47.625 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:47.625 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:47.996 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:47.997 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:47.997 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:47.998 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:48.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:48.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:48.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:48.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:48.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:48.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:48.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:48.502 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:48.502 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:49.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368368 2021-05-12T09:42:49.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368368 '!=' null 2021-05-12T09:42:49.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:49.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:49.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:49.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:49.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:49.118 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:49.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:49.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:49.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:49.120 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:49.120 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:49.490 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:49.491 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:49.492 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:49.492 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:50.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:50.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:50.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:50.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:50.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:50.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:50.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:50.023 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:50.023 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:50.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989338 2021-05-12T09:42:50.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989338 '!=' null 2021-05-12T09:42:50.522 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:50.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:50.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:50.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:50.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:50.624 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:50.624 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:50.625 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:50.625 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:50.625 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:50.625 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:51.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:51.028 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:51.029 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:51.029 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:51.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:51.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:51.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:51.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:51.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:51.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:51.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:51.531 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:51.531 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:52.085 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368344 2021-05-12T09:42:52.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368344 '!=' null 2021-05-12T09:42:52.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:52.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:52.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:52.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:52.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:52.187 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:52.188 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:52.188 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:52.188 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:52.188 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:52.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:52.559 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:52.560 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:52.560 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:52.561 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:53.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:53.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:53.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:53.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:53.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:53.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:53.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:53.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:53.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:53.571 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368344 2021-05-12T09:42:53.571 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368344 '!=' null 2021-05-12T09:42:53.572 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:53.572 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:53.572 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:53.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:53.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:53.673 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:53.673 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:53.674 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:53.674 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:53.674 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:53.674 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:54.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:54.044 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:54.045 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:54.046 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:54.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:54.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:54.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:54.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:54.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:54.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:54.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:54.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:54.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:55.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989356 2021-05-12T09:42:55.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989356 '!=' null 2021-05-12T09:42:55.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:55.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:55.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:55.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:55.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:55.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:55.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:55.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:55.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:55.165 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:55.165 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:55.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:55.533 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:55.534 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:55.534 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:56.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:56.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:56.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:56.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:56.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:56.031 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:56.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:56.032 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:56.032 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:56.555 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241969 2021-05-12T09:42:56.555 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241969 '!=' null 2021-05-12T09:42:56.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:56.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:56.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:56.656 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:56.656 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:56.658 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:56.658 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:56.658 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:56.658 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:56.659 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:56.659 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:57.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:57.036 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:57.036 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:57.037 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:57.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:57.539 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:57.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:57.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:57.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:57.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:57.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:57.541 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:57.541 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:58.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241941 2021-05-12T09:42:58.125 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241941 '!=' null 2021-05-12T09:42:58.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:58.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:58.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:58.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:58.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:58.227 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:58.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:58.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:58.228 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:58.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:58.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:42:58.594 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:42:58.596 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:42:58.596 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:42:58.597 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:42:59.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:42:59.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:42:59.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:42:59.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:42:59.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:42:59.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:42:59.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:42:59.115 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:42:59.115 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:42:59.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241941 2021-05-12T09:42:59.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241941 '!=' null 2021-05-12T09:42:59.663 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:42:59.664 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:42:59.664 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:42:59.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:42:59.765 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:42:59.766 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:42:59.766 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:42:59.766 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:42:59.766 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:42:59.767 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:42:59.767 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:00.268 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:00.269 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:00.269 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:00.269 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:00.779 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:00.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:00.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:00.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:00.780 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:00.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:00.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:00.782 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:00.782 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:01.285 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862966 2021-05-12T09:43:01.286 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862966 '!=' null 2021-05-12T09:43:01.286 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:01.286 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:01.287 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:01.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:01.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:01.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:01.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:01.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:01.389 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:01.389 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:01.389 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:01.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:01.763 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:01.763 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:01.763 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:02.261 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:02.261 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:02.261 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:02.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:02.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:02.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:02.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:02.263 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:02.263 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:02.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242001 2021-05-12T09:43:02.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242001 '!=' null 2021-05-12T09:43:02.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:02.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:02.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:02.864 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:02.864 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:02.865 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:02.865 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:02.866 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:02.866 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:02.866 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:02.866 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:03.238 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:03.239 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:03.239 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:03.240 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:03.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:03.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:03.748 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:03.748 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:03.748 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:03.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:03.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:03.750 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:03.750 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:04.269 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736661 2021-05-12T09:43:04.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736661 '!=' null 2021-05-12T09:43:04.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:04.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:04.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:04.368 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:04.369 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:04.370 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:04.370 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:04.371 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:04.371 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:04.371 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:04.371 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:04.738 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:04.739 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:04.740 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:04.740 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:05.246 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:05.246 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:05.246 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:05.247 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:05.247 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:05.247 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:05.247 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:05.248 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:05.248 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:05.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736661 2021-05-12T09:43:05.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736661 '!=' null 2021-05-12T09:43:05.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:05.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:05.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:05.850 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:05.851 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:05.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:05.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:05.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:05.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:05.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:05.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:06.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:06.224 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:06.224 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:06.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:06.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:06.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:06.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:06.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:06.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:06.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:06.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:06.728 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:06.728 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:07.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115634 2021-05-12T09:43:07.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115634 '!=' null 2021-05-12T09:43:07.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:07.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:07.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:07.331 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:07.331 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:07.332 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:07.332 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:07.332 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:07.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:07.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:07.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:07.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:07.707 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:07.707 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:07.708 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:08.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:08.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:08.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:08.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:08.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:08.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:08.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:08.224 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:08.224 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:08.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115650 2021-05-12T09:43:08.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115650 '!=' null 2021-05-12T09:43:08.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:08.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:08.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:08.826 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:08.826 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:08.830 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:08.830 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:08.831 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:08.831 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:08.831 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:08.833 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:09.207 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:09.208 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:09.209 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:09.209 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:09.731 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:09.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:09.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:09.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:09.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:09.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:09.734 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:09.734 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:09.734 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:10.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863009 2021-05-12T09:43:10.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863009 '!=' null 2021-05-12T09:43:10.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:10.302 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:10.303 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:10.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:10.402 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:10.404 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:10.404 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:10.404 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:10.404 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:10.405 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:10.405 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:10.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:10.782 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:10.783 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:10.783 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:11.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:11.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:11.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:11.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:11.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:11.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:11.284 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:11.284 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:11.284 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:11.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863009 2021-05-12T09:43:11.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863009 '!=' null 2021-05-12T09:43:11.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:11.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:11.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:11.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:11.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:11.886 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:11.886 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:11.886 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:11.886 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:11.887 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:11.887 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:12.261 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:12.261 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:12.261 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:12.261 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:12.767 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:12.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:12.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:12.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:12.768 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:12.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:12.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:12.770 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:12.770 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:13.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241991 2021-05-12T09:43:13.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241991 '!=' null 2021-05-12T09:43:13.278 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:13.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:13.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:13.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:13.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:13.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:13.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:13.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:13.381 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:13.381 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:13.381 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:13.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:13.752 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:13.752 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:13.752 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:14.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:14.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:14.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:14.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:14.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:14.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:14.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:14.265 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:14.265 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:14.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862982 2021-05-12T09:43:14.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862982 '!=' null 2021-05-12T09:43:14.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:14.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:14.782 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:14.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:14.882 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:14.883 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:14.884 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:14.884 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:14.884 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:14.884 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:14.885 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:15.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:15.256 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:15.256 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:15.256 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:15.760 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:15.760 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:15.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:15.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:15.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:15.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:15.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:15.763 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:15.764 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:16.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368327 2021-05-12T09:43:16.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368327 '!=' null 2021-05-12T09:43:16.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:16.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:16.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:16.375 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:16.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:16.377 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:16.377 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:16.377 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:16.377 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:16.378 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:16.378 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:16.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:16.749 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:16.750 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:16.750 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:17.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:17.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:17.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:17.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:17.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:17.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:17.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:17.251 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:17.251 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:17.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368327 2021-05-12T09:43:17.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368327 '!=' null 2021-05-12T09:43:17.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:17.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:17.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:17.851 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:17.851 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:17.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:17.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:17.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:17.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:17.854 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:17.854 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:18.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:18.226 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:18.226 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:18.226 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:18.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:18.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:18.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:18.735 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:18.736 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:18.736 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:18.736 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:18.737 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:18.737 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:19.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368323 2021-05-12T09:43:19.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368323 '!=' null 2021-05-12T09:43:19.251 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:19.251 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:19.252 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:19.350 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:19.351 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:19.352 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:19.353 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:19.353 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:19.353 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:19.354 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:19.354 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:19.731 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:19.732 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:19.732 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:19.733 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:20.259 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:20.259 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:20.259 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:20.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:20.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:20.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:20.260 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:20.261 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:20.261 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:20.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115680 2021-05-12T09:43:20.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115680 '!=' null 2021-05-12T09:43:20.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:20.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:20.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:20.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:20.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:20.873 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:20.873 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:20.873 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:20.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:20.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:20.875 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:21.241 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:21.243 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:21.243 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:21.244 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:21.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:21.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:21.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:21.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:21.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:21.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:21.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:21.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:21.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:22.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368318 2021-05-12T09:43:22.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368318 '!=' null 2021-05-12T09:43:22.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:22.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:22.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:22.350 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:22.350 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:22.351 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:22.352 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:22.352 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:22.352 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:22.352 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:22.353 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:22.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:22.728 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:22.729 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:22.729 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:23.243 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:23.243 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:23.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:23.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:23.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:23.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:23.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:23.245 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:23.245 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:23.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368318 2021-05-12T09:43:23.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368318 '!=' null 2021-05-12T09:43:23.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:23.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:23.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:23.850 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:23.850 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:23.851 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:23.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:23.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:23.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:23.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:23.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:24.221 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:24.223 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:24.223 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:24.223 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:24.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:24.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:24.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:24.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:24.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:24.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:24.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:24.723 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:24.723 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:25.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989337 2021-05-12T09:43:25.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989337 '!=' null 2021-05-12T09:43:25.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:25.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:25.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:25.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:25.345 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:25.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:25.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:25.347 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:25.347 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:25.347 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:25.347 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:25.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:25.715 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:25.715 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:25.716 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:26.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:26.234 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:26.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:26.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:26.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:26.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:26.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:26.237 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:26.237 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:26.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368338 2021-05-12T09:43:26.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368338 '!=' null 2021-05-12T09:43:26.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:26.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:26.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:26.843 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:26.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:26.845 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:26.845 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:26.845 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:26.845 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:26.846 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:26.846 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:27.212 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:27.213 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:27.213 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:27.214 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:27.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:27.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:27.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:27.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:27.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:27.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:27.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:27.726 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:27.726 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:28.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368341 2021-05-12T09:43:28.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368341 '!=' null 2021-05-12T09:43:28.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:28.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:28.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:28.331 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:28.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:28.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:28.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:28.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:28.333 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:28.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:28.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:28.703 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:28.703 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:28.703 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:28.704 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:29.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:29.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:29.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:29.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:29.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:29.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:29.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:29.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:29.206 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:29.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368341 2021-05-12T09:43:29.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368341 '!=' null 2021-05-12T09:43:29.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:29.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:29.707 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:29.809 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:29.811 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:29.811 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:29.811 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:29.812 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:29.812 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:29.813 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:29.814 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:30.180 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:30.181 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:30.181 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:30.181 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:30.694 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:30.695 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:30.695 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:30.695 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:30.695 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:30.695 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:30.696 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:30.696 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:30.696 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:31.199 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989315 2021-05-12T09:43:31.200 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989315 '!=' null 2021-05-12T09:43:31.200 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:31.200 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:31.200 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:31.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:31.299 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:31.300 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:31.301 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:31.301 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:31.301 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:31.301 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:31.302 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:31.672 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:31.673 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:31.673 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:31.674 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:32.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:32.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:32.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:32.193 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:32.194 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:32.194 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:32.194 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:32.195 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:32.195 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:32.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241993 2021-05-12T09:43:32.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241993 '!=' null 2021-05-12T09:43:32.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:32.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:32.747 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:32.847 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:32.847 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:32.848 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:32.848 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:32.849 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:32.849 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:32.849 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:32.849 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:33.222 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:33.223 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:33.223 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:33.223 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:33.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:33.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:33.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:33.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:33.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:33.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:33.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:33.730 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:33.730 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:34.270 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989346 2021-05-12T09:43:34.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989346 '!=' null 2021-05-12T09:43:34.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:34.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:34.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:34.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:34.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:34.373 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:34.373 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:34.373 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:34.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:34.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:34.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:34.746 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:34.747 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:34.748 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:34.748 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:35.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:35.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:35.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:35.254 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:35.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:35.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:35.255 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:35.256 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:35.256 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:35.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989346 2021-05-12T09:43:35.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989346 '!=' null 2021-05-12T09:43:35.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:35.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:35.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:35.852 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:35.852 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:35.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:35.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:35.854 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:35.854 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:35.854 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:35.854 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:36.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:36.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:36.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:36.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:36.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:36.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:36.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:36.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:36.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:36.741 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:36.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:36.743 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:36.743 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:37.240 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368333 2021-05-12T09:43:37.241 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368333 '!=' null 2021-05-12T09:43:37.241 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:37.241 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:37.241 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:37.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:37.341 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:37.343 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:37.343 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:37.343 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:37.343 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:37.344 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:37.344 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:37.713 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:37.714 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:37.714 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:37.714 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:38.216 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:38.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:38.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:38.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:38.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:38.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:38.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:38.219 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:38.219 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:38.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368335 2021-05-12T09:43:38.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368335 '!=' null 2021-05-12T09:43:38.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:38.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:38.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:38.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:38.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:38.822 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:38.823 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:38.823 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:38.823 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:38.823 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:38.824 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:39.210 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:39.211 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:39.211 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:39.211 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:39.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:39.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:39.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:39.732 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:39.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:39.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:39.733 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:39.734 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:39.735 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:40.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989305 2021-05-12T09:43:40.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989305 '!=' null 2021-05-12T09:43:40.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:40.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:40.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:40.352 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:40.352 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:40.353 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:40.353 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:40.353 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:40.353 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:40.353 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:40.354 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:40.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:40.724 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:40.725 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:40.725 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:41.235 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:41.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:41.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:41.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:41.236 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:41.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:41.237 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:41.237 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:41.238 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:41.753 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989305 2021-05-12T09:43:41.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989305 '!=' null 2021-05-12T09:43:41.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:41.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:41.755 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:41.854 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:41.854 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:41.856 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:41.856 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:41.856 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:41.856 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:41.856 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:41.857 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:42.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:42.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:42.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:42.230 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:42.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:42.725 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:42.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:42.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:42.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:42.726 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:42.727 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:42.727 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:42.728 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:43.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368362 2021-05-12T09:43:43.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368362 '!=' null 2021-05-12T09:43:43.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:43.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:43.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:43.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:43.325 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:43.326 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:43.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:43.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:43.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:43.327 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:43.328 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:43.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:43.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:43.707 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:43.707 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:44.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:44.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:44.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:44.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:44.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:44.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:44.219 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:44.220 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:44.220 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863040 2021-05-12T09:43:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863040 '!=' null 2021-05-12T09:43:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:44.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:44.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:44.821 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:44.822 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:44.822 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:44.822 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:44.823 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:44.823 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:44.823 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:45.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:45.203 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:45.203 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:45.204 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:45.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:45.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:45.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:45.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:45.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:45.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:45.701 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:45.702 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:45.702 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:46.210 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115703 2021-05-12T09:43:46.210 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115703 '!=' null 2021-05-12T09:43:46.211 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:46.211 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:46.211 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:46.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:46.311 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:46.312 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:46.312 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:46.313 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:46.313 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:46.313 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:46.313 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:46.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:46.683 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:46.684 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:46.684 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:47.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:47.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:47.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:47.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:47.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:47.185 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:47.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:47.186 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:47.187 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:47.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115703 2021-05-12T09:43:47.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115703 '!=' null 2021-05-12T09:43:47.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:47.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:47.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:47.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:47.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:47.784 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:47.785 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:47.785 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:47.785 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:47.785 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:47.786 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:48.154 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:48.154 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:48.155 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:48.155 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:48.667 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:48.667 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:48.668 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:48.668 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:48.668 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:48.668 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:48.668 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:48.669 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:48.669 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:49.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115697 2021-05-12T09:43:49.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115697 '!=' null 2021-05-12T09:43:49.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:49.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:49.174 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:49.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:49.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:49.275 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:49.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:49.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:49.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:49.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:49.277 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:49.648 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:49.649 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:49.650 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:49.650 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:50.149 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:50.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:50.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:50.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:50.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:50.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:50.151 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:50.152 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:50.152 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:50.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736695 2021-05-12T09:43:50.650 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736695 '!=' null 2021-05-12T09:43:50.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:50.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:50.651 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:50.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:50.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:50.752 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:50.752 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:50.753 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:50.753 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:50.753 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:50.753 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:51.126 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:51.127 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:51.128 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:51.129 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:51.630 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:51.630 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:51.631 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:51.631 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:51.631 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:51.631 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:51.632 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:51.632 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:51.632 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:52.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115655 2021-05-12T09:43:52.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115655 '!=' null 2021-05-12T09:43:52.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:52.132 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:52.132 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:52.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:52.232 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:52.233 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:52.233 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:52.233 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:52.234 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:52.234 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:52.234 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:52.601 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:52.602 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:52.602 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:52.602 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:53.097 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:53.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:53.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:53.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:53.098 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:53.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:53.099 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:53.099 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:53.100 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:53.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115655 2021-05-12T09:43:53.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115655 '!=' null 2021-05-12T09:43:53.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:53.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:53.600 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:53.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:53.700 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:53.701 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:53.701 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:53.701 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:53.701 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:53.702 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:53.702 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:54.111 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:54.112 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:54.113 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:54.113 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:54.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:54.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:54.692 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:54.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:54.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:54.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:54.693 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:54.694 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:54.694 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:55.282 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862987 2021-05-12T09:43:55.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862987 '!=' null 2021-05-12T09:43:55.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:55.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:55.283 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:55.383 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:55.384 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:55.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:55.386 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:55.387 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:55.387 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:55.387 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:55.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:55.756 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:55.757 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:55.757 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:55.757 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:56.258 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:56.258 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:56.258 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:56.258 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:56.259 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:56.259 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:56.259 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:56.260 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:56.260 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:56.761 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368302 2021-05-12T09:43:56.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368302 '!=' null 2021-05-12T09:43:56.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:56.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:56.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:56.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:56.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:56.864 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:56.864 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:56.864 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:56.865 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:56.865 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:56.865 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:57.251 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:57.252 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:57.253 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:57.253 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:57.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:57.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:57.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:57.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:57.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:57.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:57.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:57.764 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:57.765 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:58.275 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368288 2021-05-12T09:43:58.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368288 '!=' null 2021-05-12T09:43:58.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:58.276 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:58.277 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:58.376 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:58.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:58.378 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:58.378 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:58.378 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:58.378 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:58.379 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:58.379 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:43:58.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:43:58.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:43:58.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:43:58.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:43:59.251 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:43:59.252 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:43:59.252 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:43:59.252 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:43:59.253 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:43:59.253 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:43:59.253 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:43:59.254 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:43:59.254 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:43:59.751 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368288 2021-05-12T09:43:59.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368288 '!=' null 2021-05-12T09:43:59.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:43:59.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:43:59.752 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:43:59.852 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:43:59.852 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:43:59.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:43:59.854 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:43:59.854 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:43:59.854 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:43:59.854 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:43:59.855 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:00.249 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:00.250 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:00.251 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:00.251 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:00.762 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:00.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:00.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:00.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:00.763 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:00.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:00.764 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:00.765 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:00.765 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:01.265 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989253 2021-05-12T09:44:01.266 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989253 '!=' null 2021-05-12T09:44:01.266 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:01.266 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:01.267 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:01.366 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:01.367 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:01.368 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:01.369 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:01.369 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:01.370 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:01.370 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:01.370 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:01.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:01.743 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:01.743 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:01.743 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:02.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:02.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:02.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:02.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:02.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:02.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:02.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:02.274 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:02.274 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:02.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368270 2021-05-12T09:44:02.774 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368270 '!=' null 2021-05-12T09:44:02.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:02.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:02.775 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:02.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:02.875 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:02.876 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:02.877 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:02.877 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:02.877 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:02.877 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:02.878 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:03.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:03.274 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:03.274 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:03.275 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:03.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:03.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:03.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:03.797 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:03.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:03.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:03.798 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:03.798 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:03.798 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:04.306 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989280 2021-05-12T09:44:04.306 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989280 '!=' null 2021-05-12T09:44:04.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:04.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:04.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:04.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:04.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:04.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:04.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:04.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:04.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:04.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:04.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:04.781 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:04.781 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:04.782 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:04.782 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:05.292 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:05.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:05.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:05.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:05.293 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:05.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:05.294 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:05.294 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:05.295 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:05.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989280 2021-05-12T09:44:05.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989280 '!=' null 2021-05-12T09:44:05.827 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:05.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:05.828 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:05.927 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:05.928 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:05.929 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:05.929 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:05.929 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:05.929 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:05.930 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:05.930 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:06.303 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:06.304 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:06.305 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:06.305 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:06.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:06.805 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:06.806 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:06.806 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:06.806 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:06.807 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:06.807 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:06.807 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:06.807 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:07.306 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241918 2021-05-12T09:44:07.306 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241918 '!=' null 2021-05-12T09:44:07.306 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:07.306 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:07.307 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:07.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:07.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:07.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:07.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:07.409 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:07.410 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:07.410 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:07.411 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:07.777 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:07.778 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:07.778 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:07.779 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:08.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:08.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:08.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:08.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:08.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:08.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:08.281 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:08.282 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:08.282 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:08.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241963 2021-05-12T09:44:08.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241963 '!=' null 2021-05-12T09:44:08.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:08.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:08.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:08.884 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:08.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:08.886 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:08.886 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:08.887 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:08.887 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:08.887 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:08.887 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:09.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:09.262 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:09.263 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:09.263 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:09.769 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:09.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:09.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:09.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:09.770 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:09.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:09.771 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:09.771 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:09.772 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:10.271 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989319 2021-05-12T09:44:10.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989319 '!=' null 2021-05-12T09:44:10.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:10.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:10.272 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:10.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:10.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:10.373 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:10.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:10.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:10.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:10.374 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:10.375 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:10.742 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:10.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:10.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:10.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:11.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:11.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:11.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:11.245 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:11.246 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:11.246 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:11.246 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:11.247 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:11.247 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:11.749 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989319 2021-05-12T09:44:11.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989319 '!=' null 2021-05-12T09:44:11.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:11.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:11.750 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:11.850 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:11.850 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:11.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:11.852 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:11.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:11.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:11.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:11.853 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:12.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:12.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:12.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:12.229 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:12.737 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:12.738 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:12.738 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:12.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:12.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:12.739 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:12.740 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:12.741 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:12.741 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:13.243 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368323 2021-05-12T09:44:13.243 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368323 '!=' null 2021-05-12T09:44:13.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:13.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:13.244 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:13.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:13.344 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:13.345 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:13.345 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:13.345 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:13.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:13.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:13.346 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:13.714 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:13.715 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:13.715 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:13.716 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:14.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:14.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:14.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:14.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:14.225 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:14.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:14.226 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:14.226 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:14.227 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:14.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989338 2021-05-12T09:44:14.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989338 '!=' null 2021-05-12T09:44:14.744 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:14.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:14.745 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:14.844 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:14.845 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:14.846 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:14.846 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:14.847 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:14.847 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:14.847 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:14.847 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:15.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:15.221 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:15.221 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:15.222 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:15.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:15.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:15.721 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:15.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:15.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:15.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:15.722 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:15.723 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:15.723 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:16.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368315 2021-05-12T09:44:16.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368315 '!=' null 2021-05-12T09:44:16.227 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:16.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:16.228 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:16.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:16.328 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:16.329 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:16.329 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:16.329 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:16.330 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:16.330 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:16.330 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:16.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:16.725 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:16.725 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:16.725 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:17.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:17.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:17.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:17.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:17.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:17.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:17.224 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:17.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:17.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:17.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368315 2021-05-12T09:44:17.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368315 '!=' null 2021-05-12T09:44:17.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:17.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:17.724 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:17.823 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:17.824 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:17.825 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:17.826 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:17.826 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:17.827 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:17.827 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:17.827 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:18.195 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:18.196 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:18.197 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:18.197 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:18.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:18.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:18.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:18.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:18.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:18.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:18.699 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:18.700 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:18.700 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:19.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368315 2021-05-12T09:44:19.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368315 '!=' null 2021-05-12T09:44:19.204 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:19.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:19.205 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:19.304 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:19.305 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:19.306 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:19.306 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:19.307 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:19.307 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:19.307 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:19.307 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:19.682 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:19.682 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:19.683 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:19.683 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:20.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:20.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:20.279 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:20.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:20.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:20.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:20.280 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:20.281 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:20.281 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:20.789 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862908 2021-05-12T09:44:20.789 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862908 '!=' null 2021-05-12T09:44:20.790 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:20.790 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:20.790 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:20.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:20.890 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:20.891 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:20.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:20.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:20.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:20.892 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:20.893 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:21.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:21.265 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:21.265 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:21.266 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:21.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:21.783 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:21.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:21.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:21.784 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:21.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:21.785 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:21.785 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:21.785 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:22.285 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115590 2021-05-12T09:44:22.285 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115590 '!=' null 2021-05-12T09:44:22.285 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:22.285 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:22.286 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:22.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:22.386 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:22.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:22.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:22.389 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:22.389 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:22.390 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:22.390 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:22.756 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:22.757 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:22.758 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:22.758 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:23.262 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:23.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:23.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:23.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:23.263 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:23.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:23.264 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:23.264 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:23.265 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:23.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115590 2021-05-12T09:44:23.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115590 '!=' null 2021-05-12T09:44:23.772 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:23.773 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:23.773 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:23.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:23.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:23.875 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:23.875 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:23.875 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:23.875 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:23.875 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:23.876 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:24.265 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:24.266 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:24.266 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:24.267 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:24.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:24.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:24.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:24.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:24.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:24.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:24.874 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:24.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:24.875 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:25.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736524 2021-05-12T09:44:25.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736524 '!=' null 2021-05-12T09:44:25.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:25.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:25.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:25.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:25.579 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:25.580 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:25.580 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:25.581 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:25.581 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:25.581 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:25.581 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:25.950 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:25.951 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:25.952 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:25.953 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:26.452 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:26.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:26.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:26.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:26.453 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:26.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:26.454 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:26.454 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:26.455 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:26.981 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115578 2021-05-12T09:44:26.981 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115578 '!=' null 2021-05-12T09:44:26.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:26.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:26.982 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:27.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:27.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:27.083 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:27.083 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:27.084 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:27.084 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:27.084 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:27.085 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:27.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:27.462 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:27.463 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:27.463 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:27.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:27.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:27.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:27.997 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:27.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:27.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:27.998 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:27.999 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:27.999 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:28.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115623 2021-05-12T09:44:28.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115623 '!=' null 2021-05-12T09:44:28.498 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:28.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:28.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:28.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:28.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:28.602 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:28.602 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:28.602 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:28.602 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:28.603 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:28.603 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:28.969 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:28.970 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:28.970 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:28.970 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:29.474 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:29.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:29.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:29.475 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:29.477 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:29.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:29.478 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:29.479 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:29.479 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:30.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862965 2021-05-12T09:44:30.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862965 '!=' null 2021-05-12T09:44:30.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:30.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:30.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:30.161 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:30.162 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:30.163 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:30.163 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:30.163 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:30.163 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:30.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:30.164 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:30.536 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:30.537 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:30.537 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:30.537 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:31.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:31.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:31.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:31.041 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:31.042 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:31.042 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:31.042 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:31.043 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:31.043 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:31.555 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862965 2021-05-12T09:44:31.555 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862965 '!=' null 2021-05-12T09:44:31.555 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:31.555 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:31.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:31.655 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:31.656 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:31.657 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:31.657 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:31.657 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:31.657 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:31.658 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:31.658 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:32.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:32.034 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:32.034 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:32.034 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:32.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:32.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:32.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:32.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:32.533 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:32.534 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:32.534 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:32.535 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:32.535 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:33.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368302 2021-05-12T09:44:33.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368302 '!=' null 2021-05-12T09:44:33.038 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:33.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:33.039 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:33.138 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:33.139 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:33.140 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:33.140 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:33.140 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:33.141 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:33.141 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:33.141 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:33.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:33.510 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:33.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:33.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:34.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:34.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:34.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:34.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:34.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:34.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:34.035 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:34.035 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:34.036 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:34.629 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989303 2021-05-12T09:44:34.630 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989303 '!=' null 2021-05-12T09:44:34.630 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:34.630 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:34.630 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:34.730 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:34.730 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:34.731 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:34.731 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:34.732 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:34.732 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:34.732 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:34.732 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:35.104 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:35.105 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:35.106 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:35.106 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:35.617 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:35.617 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:35.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:35.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:35.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:35.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:35.619 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:35.619 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:35.620 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:36.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368334 2021-05-12T09:44:36.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368334 '!=' null 2021-05-12T09:44:36.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:36.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:36.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:36.217 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:36.218 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:36.219 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:36.219 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:36.219 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:36.220 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:36.220 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:36.220 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:36.594 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:36.595 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:36.596 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:36.596 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:37.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:37.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:37.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:37.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:37.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:37.093 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:37.093 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:37.094 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:37.094 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:37.598 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368334 2021-05-12T09:44:37.598 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368334 '!=' null 2021-05-12T09:44:37.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:37.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:37.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:37.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:37.698 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:37.699 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:37.700 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:37.700 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:37.700 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:37.700 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:37.701 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:38.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:38.106 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:38.106 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:38.106 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:38.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:38.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:38.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:38.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:38.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:38.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:38.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:38.611 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:38.612 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:39.172 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368327 2021-05-12T09:44:39.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368327 '!=' null 2021-05-12T09:44:39.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:39.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:39.173 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:39.273 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:39.274 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:39.275 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:39.275 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:39.275 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:39.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:39.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:39.276 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:39.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:39.643 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:39.644 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:39.644 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:40.143 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:40.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:40.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:40.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:40.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:40.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:40.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:40.146 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:40.146 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:40.652 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115640 2021-05-12T09:44:40.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115640 '!=' null 2021-05-12T09:44:40.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:40.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:40.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:40.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:40.754 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:40.755 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:40.755 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:40.756 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:40.756 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:40.756 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:40.756 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:41.124 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:41.124 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:41.125 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:41.125 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:41.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:41.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:41.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:41.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:41.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:41.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:41.626 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:41.626 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:41.626 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:42.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368306 2021-05-12T09:44:42.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368306 '!=' null 2021-05-12T09:44:42.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:42.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:42.130 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:42.229 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:42.230 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:42.231 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:42.231 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:42.231 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:42.231 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:42.232 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:42.232 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:42.600 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:42.601 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:42.601 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:42.601 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:43.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:43.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:43.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:43.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:43.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:43.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:43.102 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:43.102 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:43.102 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:43.603 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368306 2021-05-12T09:44:43.603 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368306 '!=' null 2021-05-12T09:44:43.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:43.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:43.604 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:43.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:43.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:43.705 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:43.705 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:43.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:43.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:43.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:43.707 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:44.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:44.083 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:44.083 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:44.083 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:44.612 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:44.612 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:44.612 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:44.613 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:45.119 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989346 2021-05-12T09:44:45.119 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989346 '!=' null 2021-05-12T09:44:45.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:45.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:45.120 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:45.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:45.220 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:45.221 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:45.222 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:45.222 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:45.222 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:45.222 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:45.223 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:45.591 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:45.593 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:45.593 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:45.593 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:46.091 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:46.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:46.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:46.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:46.092 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:46.093 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:46.093 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:46.093 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:46.094 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:46.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368280 2021-05-12T09:44:46.642 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368280 '!=' null 2021-05-12T09:44:46.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:46.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:46.643 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:46.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:46.743 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:46.744 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:46.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:46.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:46.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:46.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:46.745 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:47.115 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:47.116 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:47.116 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:47.117 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:47.626 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:47.626 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:47.627 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:47.627 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:47.627 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:47.627 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:47.628 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:47.628 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:47.629 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:48.130 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368281 2021-05-12T09:44:48.130 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368281 '!=' null 2021-05-12T09:44:48.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:48.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:48.131 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:48.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:48.231 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:48.232 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:48.232 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:48.233 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:48.233 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:48.233 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:48.233 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:48.602 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:48.602 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:48.603 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:48.603 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:49.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:49.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:49.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:49.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:49.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:49.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:49.118 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:49.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:49.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:49.622 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368281 2021-05-12T09:44:49.622 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368281 '!=' null 2021-05-12T09:44:49.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:49.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:49.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:49.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:49.723 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:49.724 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:49.724 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:49.725 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:49.725 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:49.725 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:49.725 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:50.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:50.104 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:50.104 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:50.104 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:50.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:50.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:50.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:50.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:50.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:50.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:50.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:50.608 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:50.608 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:51.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989278 2021-05-12T09:44:51.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989278 '!=' null 2021-05-12T09:44:51.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:51.113 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:51.114 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:51.213 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:51.214 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:51.215 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:51.215 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:51.215 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:51.216 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:51.216 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:51.216 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:51.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:51.597 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:51.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:51.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:52.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:52.108 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:52.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:52.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:52.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:52.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:52.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:52.110 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:52.110 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:52.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241930 2021-05-12T09:44:52.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241930 '!=' null 2021-05-12T09:44:52.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:52.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:52.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:52.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:52.709 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:52.710 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:52.710 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:52.710 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:52.711 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:52.711 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:52.711 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:53.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:53.084 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:53.084 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:53.085 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:53.597 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:53.597 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:53.598 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:53.598 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:53.598 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:53.598 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:53.599 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:53.599 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:53.600 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:54.149 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862950 2021-05-12T09:44:54.149 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862950 '!=' null 2021-05-12T09:44:54.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:54.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:54.150 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:54.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:54.250 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:54.251 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:54.251 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:54.252 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:54.252 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:54.252 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:54.252 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:54.630 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:54.631 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:54.631 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:54.632 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:55.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:55.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:55.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:55.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:55.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:55.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:55.129 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:55.129 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:55.129 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:55.627 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862950 2021-05-12T09:44:55.628 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862950 '!=' null 2021-05-12T09:44:55.628 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:55.628 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:55.628 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:55.728 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:55.729 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:55.730 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:55.730 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:55.730 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:55.731 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:55.731 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:55.731 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:56.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:56.102 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:56.102 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:56.102 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:56.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:56.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:56.608 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:56.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:56.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:56.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:56.609 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:56.610 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:56.610 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:57.122 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241934 2021-05-12T09:44:57.122 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241934 '!=' null 2021-05-12T09:44:57.123 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:57.123 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:57.123 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:57.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:57.223 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:57.224 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:57.224 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:57.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:57.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:57.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:57.225 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:57.597 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:57.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:57.599 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:57.599 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:58.100 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:58.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:58.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:58.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:58.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:58.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:58.102 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:58.102 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:58.103 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:44:58.605 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241953 2021-05-12T09:44:58.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241953 '!=' null 2021-05-12T09:44:58.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:44:58.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:44:58.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:44:58.703 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:44:58.704 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:44:58.705 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:44:58.705 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:44:58.705 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:44:58.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:44:58.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:44:58.706 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:44:59.073 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:44:59.074 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:44:59.075 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:44:59.075 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:44:59.581 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:44:59.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:44:59.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:44:59.582 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:44:59.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:44:59.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:44:59.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:44:59.584 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:44:59.584 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:00.102 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862946 2021-05-12T09:45:00.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862946 '!=' null 2021-05-12T09:45:00.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:00.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:00.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:00.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:00.203 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:00.204 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:00.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:00.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:00.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:00.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:00.205 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:00.580 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:00.581 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:00.582 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:00.582 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:01.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:01.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:01.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:01.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:01.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:01.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:01.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:01.084 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:01.085 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:01.615 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862946 2021-05-12T09:45:01.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862946 '!=' null 2021-05-12T09:45:01.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:01.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:01.616 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:01.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:01.716 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:01.717 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:01.718 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:01.718 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:01.718 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:01.718 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:01.719 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:02.089 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:02.090 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:02.090 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:02.090 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:02.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:02.591 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:02.591 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:02.591 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:02.592 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:02.592 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:02.592 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:02.593 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:02.593 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:03.095 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241899 2021-05-12T09:45:03.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241899 '!=' null 2021-05-12T09:45:03.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:03.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:03.096 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:03.196 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:03.332 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:03.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:03.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:03.334 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:03.335 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:03.335 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:03.335 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:03.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:03.574 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:03.574 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:03.574 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:04.080 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:04.080 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:04.081 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:04.081 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:04.081 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:04.081 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:04.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:04.082 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:04.082 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:04.583 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862919 2021-05-12T09:45:04.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862919 '!=' null 2021-05-12T09:45:04.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:04.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:04.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:04.683 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:04.684 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:04.685 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:04.685 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:04.686 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:04.686 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:04.686 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:04.686 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:05.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:05.063 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:05.063 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:05.064 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:05.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:05.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:05.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:05.561 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:05.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:05.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:05.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:05.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:05.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:06.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241930 2021-05-12T09:45:06.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241930 '!=' null 2021-05-12T09:45:06.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:06.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:06.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:06.164 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:06.164 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:06.165 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:06.165 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:06.166 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:06.166 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:06.166 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:06.166 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:06.535 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:06.536 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:06.536 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:06.537 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:07.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:07.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:07.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:07.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:07.046 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:07.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:07.047 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:07.047 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:07.048 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:07.545 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241930 2021-05-12T09:45:07.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241930 '!=' null 2021-05-12T09:45:07.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:07.546 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:07.547 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:07.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:07.646 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:07.647 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:07.647 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:07.647 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:07.648 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:07.648 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:07.648 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:08.014 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:08.015 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:08.016 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:08.016 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:08.527 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:08.528 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:08.528 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:08.528 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:08.528 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:08.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:08.529 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:08.529 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:08.530 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:09.029 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241940 2021-05-12T09:45:09.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241940 '!=' null 2021-05-12T09:45:09.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:09.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:09.030 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:09.130 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:09.130 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:09.131 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:09.131 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:09.132 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:09.132 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:09.132 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:09.132 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:09.505 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:09.506 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:09.506 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:09.507 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:10.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:10.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:10.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:10.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:10.064 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:10.065 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:10.065 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:10.066 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:10.066 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:10.566 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862956 2021-05-12T09:45:10.566 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862956 '!=' null 2021-05-12T09:45:10.566 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:10.567 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:10.567 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:10.667 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:10.667 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:10.668 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:10.668 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:10.669 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:10.669 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:10.669 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:10.669 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:11.040 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:11.041 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:11.042 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:11.043 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:11.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:11.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:11.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:11.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:11.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:11.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:11.543 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:11.544 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:11.544 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:12.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241951 2021-05-12T09:45:12.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241951 '!=' null 2021-05-12T09:45:12.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:12.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:12.045 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:12.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:12.145 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:12.146 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:12.146 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:12.147 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:12.147 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:12.147 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:12.147 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:12.537 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:12.538 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:12.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:12.539 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:13.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:13.059 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:13.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:13.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:13.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:13.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:13.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:13.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:13.061 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:13.569 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5241951 2021-05-12T09:45:13.569 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5241951 '!=' null 2021-05-12T09:45:13.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:13.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:13.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:13.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:13.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:13.672 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:13.672 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:13.672 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:13.672 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:13.673 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:13.673 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:14.071 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:14.072 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:14.072 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:14.072 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:14.572 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:14.572 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:14.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:14.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:14.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:14.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:14.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:14.574 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:14.574 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:15.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989352 2021-05-12T09:45:15.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989352 '!=' null 2021-05-12T09:45:15.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:15.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:15.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:15.183 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:15.183 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:15.184 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:15.184 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:15.185 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:15.185 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:15.185 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:15.185 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:15.551 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:15.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:15.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:15.552 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:16.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:16.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:16.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:16.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:16.054 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:16.055 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:16.055 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:16.055 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:16.055 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:16.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368346 2021-05-12T09:45:16.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368346 '!=' null 2021-05-12T09:45:16.573 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:16.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:16.574 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:16.673 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:16.674 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:16.675 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:16.675 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:16.676 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:16.676 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:16.676 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:16.676 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:17.042 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:17.043 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:17.044 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:17.044 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:17.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:17.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:17.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:17.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:17.555 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:17.555 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:17.555 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:17.556 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:17.556 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:18.076 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368349 2021-05-12T09:45:18.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368349 '!=' null 2021-05-12T09:45:18.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:18.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:18.077 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:18.176 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:18.177 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:18.178 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:18.178 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:18.178 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:18.178 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:18.179 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:18.179 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:18.548 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:18.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:18.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:18.549 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:19.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:19.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:19.052 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:19.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:19.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:19.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:19.053 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:19.054 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:19.054 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:19.557 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368349 2021-05-12T09:45:19.557 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368349 '!=' null 2021-05-12T09:45:19.557 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:19.558 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:19.558 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:19.657 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:19.658 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:19.659 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:19.660 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:19.660 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:19.660 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:19.660 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:19.661 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:20.034 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:20.034 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:20.035 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:20.035 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:20.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:20.540 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:20.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:20.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:20.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:20.541 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:20.542 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:20.542 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:20.543 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:21.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989388 2021-05-12T09:45:21.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989388 '!=' null 2021-05-12T09:45:21.043 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:21.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:21.044 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:21.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:21.144 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:21.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:21.145 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:21.146 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:21.146 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:21.146 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:21.146 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:21.514 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:21.515 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:21.515 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:21.516 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:22.015 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:22.015 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:22.015 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:22.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:22.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:22.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:22.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:22.017 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:22.017 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:22.517 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368383 2021-05-12T09:45:22.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368383 '!=' null 2021-05-12T09:45:22.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:22.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:22.518 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:22.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:22.618 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:22.619 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:22.620 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:22.620 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:22.620 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:22.621 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:22.621 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:22.995 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:22.996 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:22.996 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:22.996 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:23.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:23.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:23.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:23.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:23.500 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:23.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:23.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:23.501 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:23.501 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:24.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989436 2021-05-12T09:45:24.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989436 '!=' null 2021-05-12T09:45:24.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:24.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:24.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:24.127 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:24.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:24.129 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:24.129 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:24.129 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:24.130 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:24.130 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:24.130 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:24.505 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:24.506 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:24.507 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:24.507 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:25.010 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:25.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:25.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:25.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:25.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:25.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:25.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:25.013 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:25.013 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:25.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989436 2021-05-12T09:45:25.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989436 '!=' null 2021-05-12T09:45:25.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:25.553 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:25.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:25.653 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:25.654 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:25.655 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:25.656 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:25.656 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:25.656 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:25.656 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:25.657 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:26.024 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:26.025 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:26.026 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:26.026 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:26.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:26.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:26.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:26.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:26.531 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:26.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:26.532 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:26.533 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:26.533 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:27.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242065 2021-05-12T09:45:27.032 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242065 '!=' null 2021-05-12T09:45:27.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:27.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:27.033 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:27.132 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:27.133 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:27.134 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:27.134 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:27.134 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:27.135 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:27.135 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:27.135 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:27.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:27.510 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:27.510 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:27.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:28.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:28.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:28.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:28.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:28.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:28.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:28.013 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:28.014 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:28.014 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:28.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242063 2021-05-12T09:45:28.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242063 '!=' null 2021-05-12T09:45:28.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:28.523 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:28.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:28.623 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:28.624 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:28.625 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:28.625 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:28.625 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:28.626 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:28.626 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:28.626 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:29.001 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:29.001 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:29.002 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:29.003 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:29.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:29.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:29.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:29.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:29.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:29.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:29.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:29.510 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:29.511 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:30.015 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989416 2021-05-12T09:45:30.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989416 '!=' null 2021-05-12T09:45:30.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:30.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:30.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:30.116 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:30.117 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:30.118 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:30.118 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:30.118 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:30.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:30.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:30.119 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:30.488 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:30.489 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:30.489 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:30.489 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:30.988 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:30.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:30.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:30.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:30.989 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:30.990 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:30.990 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:30.990 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:30.991 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:31.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989416 2021-05-12T09:45:31.495 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989416 '!=' null 2021-05-12T09:45:31.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:31.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:31.496 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:31.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:31.596 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:31.597 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:31.597 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:31.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:31.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:31.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:31.598 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:32.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:32.008 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:32.009 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:32.009 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:32.506 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:32.506 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:32.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:32.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:32.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:32.507 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:32.508 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:32.508 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:32.509 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:33.011 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368384 2021-05-12T09:45:33.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368384 '!=' null 2021-05-12T09:45:33.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:33.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:33.012 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:33.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:33.112 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:33.113 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:33.114 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:33.114 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:33.114 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:33.115 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:33.115 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:33.499 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:33.499 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:33.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:33.500 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:34.008 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:34.008 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:34.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:34.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:34.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:34.010 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:34.010 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:34.011 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:34.011 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:34.509 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989413 2021-05-12T09:45:34.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989413 '!=' null 2021-05-12T09:45:34.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:34.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:34.510 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:34.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:34.610 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:34.611 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:34.611 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:34.612 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:34.612 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:34.612 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:34.612 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:34.979 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:34.980 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:34.980 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:34.981 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:35.479 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:35.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:35.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:35.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:35.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:35.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:35.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:35.482 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:35.482 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:36.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368331 2021-05-12T09:45:36.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368331 '!=' null 2021-05-12T09:45:36.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:36.005 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:36.006 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:36.105 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:36.106 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:36.107 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:36.108 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:36.108 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:36.108 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:36.108 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:36.109 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:36.544 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:36.545 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:36.545 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:36.546 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:37.101 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:37.102 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:37.102 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:37.102 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:37.102 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:37.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:37.103 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:37.104 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:37.104 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:37.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368331 2021-05-12T09:45:37.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368331 '!=' null 2021-05-12T09:45:37.606 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:37.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:37.607 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:37.705 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:37.706 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:37.708 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:37.708 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:37.709 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:37.709 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:37.710 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:37.710 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:38.082 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:38.082 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:38.083 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:38.083 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:38.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:38.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:38.584 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:38.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:38.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:38.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:38.585 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:38.586 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:38.586 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:39.086 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368339 2021-05-12T09:45:39.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368339 '!=' null 2021-05-12T09:45:39.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:39.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:39.087 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:39.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:39.186 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:39.187 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:39.188 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:39.188 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:39.188 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:39.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:39.189 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:39.557 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:39.558 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:39.559 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:39.559 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:40.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:40.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:40.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:40.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:40.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:40.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:40.063 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:40.064 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:40.064 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:40.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7862952 2021-05-12T09:45:40.591 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7862952 '!=' null 2021-05-12T09:45:40.591 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:40.591 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:40.592 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:40.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:40.691 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:40.693 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:40.693 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:40.693 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:40.693 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:40.693 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:40.694 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:41.071 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:41.072 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:41.072 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:41.072 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:41.570 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:41.571 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:41.571 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:41.571 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:41.571 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:41.572 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:41.572 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:41.573 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:41.573 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:42.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115628 2021-05-12T09:45:42.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115628 '!=' null 2021-05-12T09:45:42.083 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:42.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:42.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:42.183 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:42.184 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:42.185 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:42.185 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:42.185 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:42.186 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:42.186 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:42.186 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:42.554 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:42.555 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:42.555 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:42.555 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:43.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:43.060 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:43.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:43.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:43.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:43.061 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:43.062 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:43.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:43.062 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:43.555 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115628 2021-05-12T09:45:43.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115628 '!=' null 2021-05-12T09:45:43.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:43.556 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:43.557 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:43.655 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:43.656 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:43.657 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:43.657 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:43.657 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:43.658 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:43.658 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:43.658 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:44.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:44.027 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:44.028 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:44.028 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:44.526 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:44.527 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:45.027 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=8736601 2021-05-12T09:45:45.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 8736601 '!=' null 2021-05-12T09:45:45.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:45.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:45.028 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:45.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:45.128 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:45.129 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:45.130 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:45.130 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:45.130 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:45.131 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:45.131 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:45.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:45.502 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:45.502 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:45.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:46.006 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:46.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:46.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:46.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:46.007 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:46.008 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:46.008 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:46.008 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:46.009 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:46.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115618 2021-05-12T09:45:46.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115618 '!=' null 2021-05-12T09:45:46.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:46.525 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:46.526 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:46.625 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:46.626 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:46.627 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:46.627 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:46.627 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:46.627 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:46.628 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:46.628 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:46.999 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:46.999 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:47.000 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:47.000 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:47.503 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:47.504 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:47.504 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:47.504 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:47.504 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:47.504 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:47.505 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:47.505 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:47.506 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:48.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115687 2021-05-12T09:45:48.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115687 '!=' null 2021-05-12T09:45:48.009 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:48.010 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:48.010 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:48.109 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:48.110 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:48.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:48.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:48.111 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:48.112 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:48.112 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:48.112 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:48.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:48.481 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:48.481 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:48.481 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:48.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:48.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:48.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:48.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:48.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:48.985 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:48.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:48.986 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:48.987 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:49.487 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115687 2021-05-12T09:45:49.488 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115687 '!=' null 2021-05-12T09:45:49.488 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:49.488 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:49.488 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:49.588 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:49.589 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:49.590 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:49.590 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:49.590 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:49.590 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:49.591 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:49.591 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:49.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:49.968 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:49.969 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:49.969 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:50.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:50.469 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:50.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:50.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:50.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:50.470 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:50.471 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:50.471 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:50.472 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:50.974 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863017 2021-05-12T09:45:50.975 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863017 '!=' null 2021-05-12T09:45:50.975 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:50.975 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:50.976 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:51.075 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:51.075 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:51.076 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:51.077 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:51.077 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:51.077 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:51.077 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:51.078 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:51.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:51.443 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:51.443 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:51.443 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:51.958 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:51.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:51.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:51.959 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:51.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:51.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:51.960 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:51.961 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:51.961 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:52.489 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368376 2021-05-12T09:45:52.490 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368376 '!=' null 2021-05-12T09:45:52.490 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:52.490 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:52.490 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:52.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:52.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:52.591 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:52.592 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:52.592 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:52.592 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:52.592 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:52.593 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:52.962 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:52.962 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:52.963 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:52.963 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:53.456 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:53.456 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:53.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:53.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:53.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:53.457 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:53.458 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:53.458 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:53.459 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:53.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989387 2021-05-12T09:45:53.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989387 '!=' null 2021-05-12T09:45:53.983 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:53.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:53.984 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:54.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:54.084 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:54.085 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:54.085 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:54.086 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:54.086 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:54.086 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:54.086 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:54.456 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:54.457 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:54.458 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:54.458 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:54.971 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:54.971 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:54.971 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:54.971 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:54.972 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:54.972 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:54.972 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:54.973 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:54.973 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:55.489 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989387 2021-05-12T09:45:55.490 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989387 '!=' null 2021-05-12T09:45:55.490 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:55.490 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:55.490 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:55.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:55.590 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:55.591 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:55.592 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:55.592 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:55.592 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:55.592 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:55.593 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:55.963 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:55.964 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:55.965 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:55.965 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:56.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:56.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:56.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:56.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:56.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:56.463 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:56.464 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:56.464 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:56.464 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:56.967 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368374 2021-05-12T09:45:56.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368374 '!=' null 2021-05-12T09:45:56.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:56.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:56.968 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:57.068 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:57.069 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:57.070 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:57.070 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:57.070 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:57.070 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:57.071 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:57.071 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:57.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:57.441 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:57.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:57.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:57.939 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:57.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:57.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:57.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:57.940 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:57.941 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:57.941 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:57.941 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:57.942 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:58.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368361 2021-05-12T09:45:58.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368361 '!=' null 2021-05-12T09:45:58.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:58.461 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:58.462 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:45:58.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:45:58.562 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:45:58.563 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:45:58.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:45:58.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:45:58.564 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:45:58.565 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:45:58.565 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:45:58.936 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:45:58.936 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:45:58.937 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:45:58.937 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:45:59.440 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:45:59.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:45:59.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:45:59.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:45:59.441 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:45:59.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:45:59.442 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:45:59.442 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:45:59.443 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:45:59.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368361 2021-05-12T09:45:59.946 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368361 '!=' null 2021-05-12T09:45:59.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:45:59.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:45:59.947 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:46:00.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:46:00.048 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:46:00.048 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:46:00.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:46:00.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:46:00.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:46:00.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:46:00.049 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:46:00.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:46:00.419 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:46:00.420 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:46:00.420 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:46:00.929 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:46:00.930 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:46:00.930 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:46:00.930 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:46:00.930 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:46:00.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:46:00.931 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:46:00.932 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:46:00.932 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:46:01.429 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6115711 2021-05-12T09:46:01.429 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6115711 '!=' null 2021-05-12T09:46:01.429 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:46:01.430 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:46:01.430 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:46:01.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:46:01.530 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:46:01.531 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:46:01.531 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:46:01.531 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:46:01.532 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:46:01.532 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:46:01.532 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:46:01.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:46:01.898 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:46:01.899 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:46:01.900 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:46:02.412 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:46:02.413 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:46:02.413 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:46:02.413 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:46:02.413 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:46:02.414 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:46:02.414 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:46:02.414 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:46:02.414 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:46:02.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368347 2021-05-12T09:46:02.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368347 '!=' null 2021-05-12T09:46:02.916 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:46:02.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:46:02.917 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:46:03.016 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:46:03.017 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:46:03.018 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:46:03.018 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:46:03.018 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:46:03.019 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:46:03.019 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:46:03.019 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:46:03.387 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:46:03.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:46:03.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:46:03.388 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:46:03.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:46:03.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:46:03.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:46:03.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:46:03.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:46:03.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:46:03.897 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:46:03.897 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:46:03.897 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:46:04.399 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989381 2021-05-12T09:46:04.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989381 '!=' null 2021-05-12T09:46:04.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:46:04.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:46:04.400 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:46:04.501 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:46:04.502 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:46:04.502 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:46:04.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:46:04.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:46:04.503 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:46:04.504 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:46:04.504 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:46:04.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:46:04.873 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:46:04.873 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:46:04.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:46:05.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:46:05.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:46:05.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:46:05.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:46:05.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:46:05.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:46:05.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:46:05.381 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:46:05.382 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:46:05.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989381 2021-05-12T09:46:05.903 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989381 '!=' null 2021-05-12T09:46:05.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:46:05.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:46:05.904 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:46:06.004 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:46:06.004 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:46:06.005 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:46:06.006 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:46:06.006 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:46:06.006 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:46:06.006 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:46:06.007 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:46:06.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:46:06.372 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:46:06.373 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:46:06.373 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:46:06.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:46:06.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:46:06.872 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:46:06.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:46:06.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:46:06.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:46:06.873 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:46:06.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:46:06.874 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:46:07.388 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368374 2021-05-12T09:46:07.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368374 '!=' null 2021-05-12T09:46:07.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:46:07.389 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:46:07.390 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:46:07.488 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:46:07.488 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:46:07.489 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:46:07.489 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:46:07.490 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:46:07.490 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:46:07.490 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:46:07.490 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:46:07.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:46:07.863 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:46:07.864 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:46:07.864 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:46:08.370 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:46:08.370 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:46:08.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:46:08.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:46:08.371 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:46:08.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:46:08.372 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:46:08.372 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:46:08.372 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:46:08.885 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=4368386 2021-05-12T09:46:08.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 4368386 '!=' null 2021-05-12T09:46:08.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:46:08.886 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:46:08.887 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:46:08.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:46:08.986 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:46:08.988 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:46:08.988 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:46:08.988 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:46:08.988 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:46:08.989 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:46:08.989 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:46:09.354 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:46:09.355 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:46:09.355 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:46:09.355 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:46:09.855 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:46:09.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:46:09.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:46:09.856 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:46:09.857 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:46:09.857 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:46:09.857 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:46:09.858 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:46:09.858 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:46:10.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989388 2021-05-12T09:46:10.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989388 '!=' null 2021-05-12T09:46:10.380 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:46:10.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:46:10.381 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:46:10.480 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:46:10.481 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:46:10.482 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:46:10.482 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:46:10.483 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:46:10.483 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:46:10.483 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:46:10.484 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:46:10.854 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:46:10.856 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:46:10.856 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:46:10.857 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:46:11.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:46:11.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:46:11.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:46:11.356 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:46:11.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:46:11.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:46:11.357 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:46:11.358 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:46:11.358 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:46:11.880 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=6989388 2021-05-12T09:46:11.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 6989388 '!=' null 2021-05-12T09:46:11.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:46:11.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:46:11.881 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:46:11.981 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:46:11.981 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:46:11.983 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:46:11.983 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:46:11.983 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:46:11.983 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:46:11.984 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:46:11.984 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:46:12.359 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:46:12.360 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:46:12.361 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:46:12.361 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:46:12.862 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:46:12.864 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:46:12.864 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:46:12.864 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:46:12.864 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:46:12.865 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:46:12.865 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:46:12.865 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:46:12.866 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:46:13.392 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242013 2021-05-12T09:46:13.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242013 '!=' null 2021-05-12T09:46:13.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:46:13.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:46:13.394 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:46:13.491 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:46:13.491 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:46:13.492 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:46:13.493 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:46:13.493 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:46:13.493 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:46:13.494 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:46:13.494 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:46:13.864 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:46:13.865 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:46:13.865 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:46:13.865 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:46:14.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:46:14.377 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:46:14.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:46:14.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:46:14.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:46:14.378 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:46:14.379 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:46:14.379 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:46:14.380 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:46:14.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=7863050 2021-05-12T09:46:14.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 7863050 '!=' null 2021-05-12T09:46:14.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:46:14.921 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:46:14.922 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:46:15.021 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:46:15.022 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:46:15.023 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:46:15.023 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:46:15.024 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:46:15.024 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:46:15.024 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:46:15.024 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:46:15.393 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:46:15.394 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:46:15.394 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:46:15.395 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:46:15.894 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:46:15.894 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:46:15.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:46:15.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:46:15.895 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:46:15.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:46:15.896 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:46:15.897 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:46:15.898 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:46:16.414 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242024 2021-05-12T09:46:16.414 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242024 '!=' null 2021-05-12T09:46:16.414 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:46:16.415 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:46:16.415 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:46:16.514 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:46:16.515 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:46:16.516 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:46:16.516 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:46:16.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:46:16.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:46:16.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:46:16.517 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:46:16.888 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:46:16.889 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:46:16.889 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:46:16.889 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:46:17.418 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 = 2 2021-05-12T09:46:17.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: test 0 '!=' 0 2021-05-12T09:46:17.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: get_is_making_recovery_progress 2021-05-12T09:46:17.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: local recovery_progress 2021-05-12T09:46:17.419 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-05-12T09:46:17.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-05-12T09:46:17.420 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-05-12T09:46:17.420 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: ceph --format json status 2021-05-12T09:46:17.421 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-05-12T09:46:17.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: local progress=5242024 2021-05-12T09:46:17.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: test 5242024 '!=' null 2021-05-12T09:46:17.919 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-05-12T09:46:17.920 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: eval 2021-05-12T09:46:17.920 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: sleep 0.1 2021-05-12T09:46:18.019 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: loop+=1 2021-05-12T09:46:18.020 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1592: wait_for_clean: true 2021-05-12T09:46:18.021 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_active_clean 2021-05-12T09:46:18.021 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: local expression 2021-05-12T09:46:18.021 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-05-12T09:46:18.022 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-05-12T09:46:18.022 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: ceph --format json pg dump pgs 2021-05-12T09:46:18.022 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-05-12T09:46:18.407 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: cur_active_clean=0 2021-05-12T09:46:18.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: get_num_pgs 2021-05-12T09:46:18.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: ceph --format json status 2021-05-12T09:46:18.408 INFO:tasks.workunit.client.0.smithi028.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_pgs: jq .pgmap.num_pgs 2021-05-12T09:46:18.732 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-05-12T09:46:18.733 DEBUG:teuthology.task.console_log:Killing console logger for smithi028 2021-05-12T09:46:18.734 DEBUG:teuthology.exit:Finished running handlers